Quantifying Local Model Validity using Active Learning

Sven Lämmle, Can Bogoclu, Robert Vosshall, Anselm Haselhoff, Dirk Roos
Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, PMLR 244:2113-2135, 2024.

Abstract

Real-world applications of machine learning models are often subject to legal or policy-based regulations. Some of these regulations require ensuring the validity of the model, i.e., the approximation error being smaller than a threshold. A global metric is generally too insensitive to determine the validity of a specific prediction, whereas evaluating local validity is costly since it requires gathering additional data. We propose learning the model error to acquire a local validity estimate while reducing the amount of required data through active learning. Using model validation benchmarks, we provide empirical evidence that the proposed method can lead to an error model with sufficient discriminative properties using a relatively small amount of data. Furthermore, an increased sensitivity to local changes of the validity bounds compared to alternative approaches is demonstrated.

Cite this Paper

BibTeX
@InProceedings{pmlr-v244-lammle24a, title = {Quantifying Local Model Validity using Active Learning}, author = {L\"ammle, Sven and Bogoclu, Can and Vosshall, Robert and Haselhoff, Anselm and Roos, Dirk}, booktitle = {Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence}, pages = {2113--2135}, year = {2024}, editor = {Kiyavash, Negar and Mooij, Joris M.}, volume = {244}, series = {Proceedings of Machine Learning Research}, month = {15--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v244/main/assets/lammle24a/lammle24a.pdf}, url = {https://proceedings.mlr.press/v244/lammle24a.html}, abstract = {Real-world applications of machine learning models are often subject to legal or policy-based regulations. Some of these regulations require ensuring the validity of the model, i.e., the approximation error being smaller than a threshold. A global metric is generally too insensitive to determine the validity of a specific prediction, whereas evaluating local validity is costly since it requires gathering additional data. We propose learning the model error to acquire a local validity estimate while reducing the amount of required data through active learning. Using model validation benchmarks, we provide empirical evidence that the proposed method can lead to an error model with sufficient discriminative properties using a relatively small amount of data. Furthermore, an increased sensitivity to local changes of the validity bounds compared to alternative approaches is demonstrated.} }
Endnote
%0 Conference Paper %T Quantifying Local Model Validity using Active Learning %A Sven Lämmle %A Can Bogoclu %A Robert Vosshall %A Anselm Haselhoff %A Dirk Roos %B Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2024 %E Negar Kiyavash %E Joris M. Mooij %F pmlr-v244-lammle24a %I PMLR %P 2113--2135 %U https://proceedings.mlr.press/v244/lammle24a.html %V 244 %X Real-world applications of machine learning models are often subject to legal or policy-based regulations. Some of these regulations require ensuring the validity of the model, i.e., the approximation error being smaller than a threshold. A global metric is generally too insensitive to determine the validity of a specific prediction, whereas evaluating local validity is costly since it requires gathering additional data. We propose learning the model error to acquire a local validity estimate while reducing the amount of required data through active learning. Using model validation benchmarks, we provide empirical evidence that the proposed method can lead to an error model with sufficient discriminative properties using a relatively small amount of data. Furthermore, an increased sensitivity to local changes of the validity bounds compared to alternative approaches is demonstrated.
APA
Lämmle, S., Bogoclu, C., Vosshall, R., Haselhoff, A. & Roos, D.. (2024). Quantifying Local Model Validity using Active Learning. Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 244:2113-2135 Available from https://proceedings.mlr.press/v244/lammle24a.html.

Related Material