Formally Exploring Visual Anomaly Detection Evaluation Metrics

Nasar Iqbal, Dennis Wagner, Philipp Liznerski, Nabeel Hussain Syed, Sophie Fellenz, Niki Martinel, Marius Kloft
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:49822-49863, 2026.

Abstract

Inaccurate Visual Anomaly Detection (VAD) can lead to critical failures in safety-sensitive domains, including autonomous navigation and industrial surveillance. With the increasing abundance and rapid proliferation of VAD algorithms, their reliable evaluation has become increasingly important and challenging. Commonly used evaluation metrics often fail to capture practically relevant aspects of model behavior, yielding inconsistent or misleading assessments by overlooking errors such as redundant detections and the spatial distribution of false positives. In this paper, we formalize the requirements for VAD evaluation by introducing a set of axiomatic, verifiable properties that an evaluation metric should satisfy. Through a systematic analysis of state-of-the-art evaluation methods, we show that none satisfies all proposed properties. To address this gap, we introduce SAAM-ALARM, a novel evaluation metric that satisfies these properties. Our results show that SAAM-ALARM provides a more nuanced and theoretically sound assessment, establishing a stronger standard for performance benchmarking in VAD.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-iqbal26a, title = {Formally Exploring Visual Anomaly Detection Evaluation Metrics}, author = {Iqbal, Nasar and Wagner, Dennis and Liznerski, Philipp and Syed, Nabeel Hussain and Fellenz, Sophie and Martinel, Niki and Kloft, Marius}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {49822--49863}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/iqbal26a/iqbal26a.pdf}, url = {https://proceedings.mlr.press/v306/iqbal26a.html}, abstract = {Inaccurate Visual Anomaly Detection (VAD) can lead to critical failures in safety-sensitive domains, including autonomous navigation and industrial surveillance. With the increasing abundance and rapid proliferation of VAD algorithms, their reliable evaluation has become increasingly important and challenging. Commonly used evaluation metrics often fail to capture practically relevant aspects of model behavior, yielding inconsistent or misleading assessments by overlooking errors such as redundant detections and the spatial distribution of false positives. In this paper, we formalize the requirements for VAD evaluation by introducing a set of axiomatic, verifiable properties that an evaluation metric should satisfy. Through a systematic analysis of state-of-the-art evaluation methods, we show that none satisfies all proposed properties. To address this gap, we introduce SAAM-ALARM, a novel evaluation metric that satisfies these properties. Our results show that SAAM-ALARM provides a more nuanced and theoretically sound assessment, establishing a stronger standard for performance benchmarking in VAD.} }
Endnote
%0 Conference Paper %T Formally Exploring Visual Anomaly Detection Evaluation Metrics %A Nasar Iqbal %A Dennis Wagner %A Philipp Liznerski %A Nabeel Hussain Syed %A Sophie Fellenz %A Niki Martinel %A Marius Kloft %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-iqbal26a %I PMLR %P 49822--49863 %U https://proceedings.mlr.press/v306/iqbal26a.html %V 306 %X Inaccurate Visual Anomaly Detection (VAD) can lead to critical failures in safety-sensitive domains, including autonomous navigation and industrial surveillance. With the increasing abundance and rapid proliferation of VAD algorithms, their reliable evaluation has become increasingly important and challenging. Commonly used evaluation metrics often fail to capture practically relevant aspects of model behavior, yielding inconsistent or misleading assessments by overlooking errors such as redundant detections and the spatial distribution of false positives. In this paper, we formalize the requirements for VAD evaluation by introducing a set of axiomatic, verifiable properties that an evaluation metric should satisfy. Through a systematic analysis of state-of-the-art evaluation methods, we show that none satisfies all proposed properties. To address this gap, we introduce SAAM-ALARM, a novel evaluation metric that satisfies these properties. Our results show that SAAM-ALARM provides a more nuanced and theoretically sound assessment, establishing a stronger standard for performance benchmarking in VAD.
APA
Iqbal, N., Wagner, D., Liznerski, P., Syed, N.H., Fellenz, S., Martinel, N. & Kloft, M.. (2026). Formally Exploring Visual Anomaly Detection Evaluation Metrics. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:49822-49863 Available from https://proceedings.mlr.press/v306/iqbal26a.html.

Related Material