PAGER: Accurate Failure Characterization in Deep Regression Models

Jayaraman J. Thiagarajan, Vivek Narayanaswamy, Puja Trivedi, Rushil Anirudh
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:21069-21082, 2024.

Abstract

Safe deployment of AI models requires proactive detection of failures to prevent costly errors. To this end, we study the important problem of detecting failures in deep regression models. Existing approaches rely on epistemic uncertainty estimates or inconsistency w.r.t the training data to identify failure. Interestingly, we find that while uncertainties are necessary they are insufficient to accurately characterize failure in practice. Hence, we introduce PAGER (Principled Analysis of Generalization Errors in Regressors), a framework to systematically detect and characterize failures in deep regressors. Built upon the principle of anchored training in deep models, PAGER unifies both epistemic uncertainty and complementary manifold non-conformity scores to accurately organize samples into different risk regimes.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-j-thiagarajan24a, title = {{PAGER}: Accurate Failure Characterization in Deep Regression Models}, author = {J. Thiagarajan, Jayaraman and Narayanaswamy, Vivek and Trivedi, Puja and Anirudh, Rushil}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {21069--21082}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/j-thiagarajan24a/j-thiagarajan24a.pdf}, url = {https://proceedings.mlr.press/v235/j-thiagarajan24a.html}, abstract = {Safe deployment of AI models requires proactive detection of failures to prevent costly errors. To this end, we study the important problem of detecting failures in deep regression models. Existing approaches rely on epistemic uncertainty estimates or inconsistency w.r.t the training data to identify failure. Interestingly, we find that while uncertainties are necessary they are insufficient to accurately characterize failure in practice. Hence, we introduce PAGER (Principled Analysis of Generalization Errors in Regressors), a framework to systematically detect and characterize failures in deep regressors. Built upon the principle of anchored training in deep models, PAGER unifies both epistemic uncertainty and complementary manifold non-conformity scores to accurately organize samples into different risk regimes.} }
Endnote
%0 Conference Paper %T PAGER: Accurate Failure Characterization in Deep Regression Models %A Jayaraman J. Thiagarajan %A Vivek Narayanaswamy %A Puja Trivedi %A Rushil Anirudh %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-j-thiagarajan24a %I PMLR %P 21069--21082 %U https://proceedings.mlr.press/v235/j-thiagarajan24a.html %V 235 %X Safe deployment of AI models requires proactive detection of failures to prevent costly errors. To this end, we study the important problem of detecting failures in deep regression models. Existing approaches rely on epistemic uncertainty estimates or inconsistency w.r.t the training data to identify failure. Interestingly, we find that while uncertainties are necessary they are insufficient to accurately characterize failure in practice. Hence, we introduce PAGER (Principled Analysis of Generalization Errors in Regressors), a framework to systematically detect and characterize failures in deep regressors. Built upon the principle of anchored training in deep models, PAGER unifies both epistemic uncertainty and complementary manifold non-conformity scores to accurately organize samples into different risk regimes.
APA
J. Thiagarajan, J., Narayanaswamy, V., Trivedi, P. & Anirudh, R.. (2024). PAGER: Accurate Failure Characterization in Deep Regression Models. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:21069-21082 Available from https://proceedings.mlr.press/v235/j-thiagarajan24a.html.

Related Material