Venn-Abers Testing of Exchangeability

Ilia Nouretdinov
Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 266:43-62, 2025.

Abstract

A recurrent problem in many domains is the accurate and rapid detection of a change in the distribution of observed variables. This is important since our algorithms have been trained for a certain data distribution, and if the distribution has changed, the results will not be accurate and/or valid any longer. Instances of this problem, which are generally referred to as change-point detection, are found in fault detection in vehicle control systems, detection of the onset of an epidemic, and many other applications. Recently, new methods based on reliable machine learning have shown important advantages of this statistical task. Conformal Test Martingales (CTM) allow one to avoid this limitation and obtain valid results without information about used distributions. This is done with the assumption that the data are i.i.d. (or exchangeable) in online mode, and the corresponding martingale accumulates evidence against this assumption. This work aims to extend the conformal framework and consider the other family of reliable machine learning methods, the Venn-Abers method of probabilistic prediction, to test the data for change points. This work shows how Venn-Abers testing of exchangeability (VATE) can be founded on the ground of $e$-value theory, including recently developed $e$-pseudomartingales, and studies its advantages and drawbacks, compared to CTM. Our conclusion is that the efficiency of this approach is related to the type of causality in the data set.

Cite this Paper

BibTeX
@InProceedings{pmlr-v266-nouretdinov25a, title = {Venn-Abers Testing of Exchangeability}, author = {Nouretdinov, Ilia}, booktitle = {Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications}, pages = {43--62}, year = {2025}, editor = {Nguyen, Khuong An and Luo, Zhiyuan and Papadopoulos, Harris and Löfström, Tuwe and Carlsson, Lars and Boström, Henrik}, volume = {266}, series = {Proceedings of Machine Learning Research}, month = {10--12 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v266/main/assets/nouretdinov25a/nouretdinov25a.pdf}, url = {https://proceedings.mlr.press/v266/nouretdinov25a.html}, abstract = {A recurrent problem in many domains is the accurate and rapid detection of a change in the distribution of observed variables. This is important since our algorithms have been trained for a certain data distribution, and if the distribution has changed, the results will not be accurate and/or valid any longer. Instances of this problem, which are generally referred to as change-point detection, are found in fault detection in vehicle control systems, detection of the onset of an epidemic, and many other applications. Recently, new methods based on reliable machine learning have shown important advantages of this statistical task. Conformal Test Martingales (CTM) allow one to avoid this limitation and obtain valid results without information about used distributions. This is done with the assumption that the data are i.i.d. (or exchangeable) in online mode, and the corresponding martingale accumulates evidence against this assumption. This work aims to extend the conformal framework and consider the other family of reliable machine learning methods, the Venn-Abers method of probabilistic prediction, to test the data for change points. This work shows how Venn-Abers testing of exchangeability (VATE) can be founded on the ground of $e$-value theory, including recently developed $e$-pseudomartingales, and studies its advantages and drawbacks, compared to CTM. Our conclusion is that the efficiency of this approach is related to the type of causality in the data set.} }
Endnote
%0 Conference Paper %T Venn-Abers Testing of Exchangeability %A Ilia Nouretdinov %B Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications %C Proceedings of Machine Learning Research %D 2025 %E Khuong An Nguyen %E Zhiyuan Luo %E Harris Papadopoulos %E Tuwe Löfström %E Lars Carlsson %E Henrik Boström %F pmlr-v266-nouretdinov25a %I PMLR %P 43--62 %U https://proceedings.mlr.press/v266/nouretdinov25a.html %V 266 %X A recurrent problem in many domains is the accurate and rapid detection of a change in the distribution of observed variables. This is important since our algorithms have been trained for a certain data distribution, and if the distribution has changed, the results will not be accurate and/or valid any longer. Instances of this problem, which are generally referred to as change-point detection, are found in fault detection in vehicle control systems, detection of the onset of an epidemic, and many other applications. Recently, new methods based on reliable machine learning have shown important advantages of this statistical task. Conformal Test Martingales (CTM) allow one to avoid this limitation and obtain valid results without information about used distributions. This is done with the assumption that the data are i.i.d. (or exchangeable) in online mode, and the corresponding martingale accumulates evidence against this assumption. This work aims to extend the conformal framework and consider the other family of reliable machine learning methods, the Venn-Abers method of probabilistic prediction, to test the data for change points. This work shows how Venn-Abers testing of exchangeability (VATE) can be founded on the ground of $e$-value theory, including recently developed $e$-pseudomartingales, and studies its advantages and drawbacks, compared to CTM. Our conclusion is that the efficiency of this approach is related to the type of causality in the data set.
APA
Nouretdinov, I.. (2025). Venn-Abers Testing of Exchangeability. Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications, in Proceedings of Machine Learning Research 266:43-62 Available from https://proceedings.mlr.press/v266/nouretdinov25a.html.

Related Material