Incremental Uncertainty-aware Performance Monitoring with Active Labeling Intervention

Alexander Koebler, Thomas Decker, Ingo Thon, Volker Tresp, Florian Buettner
Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, PMLR 258:2188-2196, 2025.

Abstract

We study the problem of monitoring machine learning models under gradual distribution shifts, where circumstances change slowly over time, often leading to unnoticed yet significant declines in accuracy. To address this, we propose Incremental Uncertainty-aware Performance Monitoring (IUPM), a novel label-free method that estimates performance changes by modeling gradual shifts using optimal transport. In addition, IUPM quantifies the uncertainty in the performance prediction and introduces an active labeling procedure to restore a reliable estimate under a limited labeling budget. Our experiments show that IUPM outperforms existing performance estimation baselines in various gradual shift scenarios and that its uncertainty awareness guides label acquisition more effectively compared to other strategies.

Cite this Paper

BibTeX
@InProceedings{pmlr-v258-koebler25a, title = {Incremental Uncertainty-aware Performance Monitoring with Active Labeling Intervention}, author = {Koebler, Alexander and Decker, Thomas and Thon, Ingo and Tresp, Volker and Buettner, Florian}, booktitle = {Proceedings of The 28th International Conference on Artificial Intelligence and Statistics}, pages = {2188--2196}, year = {2025}, editor = {Li, Yingzhen and Mandt, Stephan and Agrawal, Shipra and Khan, Emtiyaz}, volume = {258}, series = {Proceedings of Machine Learning Research}, month = {03--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v258/main/assets/koebler25a/koebler25a.pdf}, url = {https://proceedings.mlr.press/v258/koebler25a.html}, abstract = {We study the problem of monitoring machine learning models under gradual distribution shifts, where circumstances change slowly over time, often leading to unnoticed yet significant declines in accuracy. To address this, we propose Incremental Uncertainty-aware Performance Monitoring (IUPM), a novel label-free method that estimates performance changes by modeling gradual shifts using optimal transport. In addition, IUPM quantifies the uncertainty in the performance prediction and introduces an active labeling procedure to restore a reliable estimate under a limited labeling budget. Our experiments show that IUPM outperforms existing performance estimation baselines in various gradual shift scenarios and that its uncertainty awareness guides label acquisition more effectively compared to other strategies.} }
Endnote
%0 Conference Paper %T Incremental Uncertainty-aware Performance Monitoring with Active Labeling Intervention %A Alexander Koebler %A Thomas Decker %A Ingo Thon %A Volker Tresp %A Florian Buettner %B Proceedings of The 28th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2025 %E Yingzhen Li %E Stephan Mandt %E Shipra Agrawal %E Emtiyaz Khan %F pmlr-v258-koebler25a %I PMLR %P 2188--2196 %U https://proceedings.mlr.press/v258/koebler25a.html %V 258 %X We study the problem of monitoring machine learning models under gradual distribution shifts, where circumstances change slowly over time, often leading to unnoticed yet significant declines in accuracy. To address this, we propose Incremental Uncertainty-aware Performance Monitoring (IUPM), a novel label-free method that estimates performance changes by modeling gradual shifts using optimal transport. In addition, IUPM quantifies the uncertainty in the performance prediction and introduces an active labeling procedure to restore a reliable estimate under a limited labeling budget. Our experiments show that IUPM outperforms existing performance estimation baselines in various gradual shift scenarios and that its uncertainty awareness guides label acquisition more effectively compared to other strategies.
APA
Koebler, A., Decker, T., Thon, I., Tresp, V. & Buettner, F.. (2025). Incremental Uncertainty-aware Performance Monitoring with Active Labeling Intervention. Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 258:2188-2196 Available from https://proceedings.mlr.press/v258/koebler25a.html.

Related Material