Evaluating Unsupervised Denoising Requires Unsupervised Metrics

Adria Marcos Morales, Matan Leibovich, Sreyas Mohan, Joshua Lawrence Vincent, Piyush Haluai, Mai Tan, Peter Crozier, Carlos Fernandez-Granda
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:23937-23957, 2023.

Abstract

Unsupervised denoising is a crucial challenge in real-world imaging applications. Unsupervised deep-learning methods have demonstrated impressive performance on benchmarks based on synthetic noise. However, no metrics exist to evaluate these methods in an unsupervised fashion. This is highly problematic for the many practical applications where ground-truth clean images are not available. In this work, we propose two novel metrics: the unsupervised mean squared error (MSE) and the unsupervised peak signal-to-noise ratio (PSNR), which are computed using only noisy data. We provide a theoretical analysis of these metrics, showing that they are asymptotically consistent estimators of the supervised MSE and PSNR. Controlled numerical experiments with synthetic noise confirm that they provide accurate approximations in practice. We validate our approach on real-world data from two imaging modalities: videos in raw format and transmission electron microscopy. Our results demonstrate that the proposed metrics enable unsupervised evaluation of denoising methods based exclusively on noisy data.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-marcos-morales23a, title = {Evaluating Unsupervised Denoising Requires Unsupervised Metrics}, author = {Marcos Morales, Adria and Leibovich, Matan and Mohan, Sreyas and Vincent, Joshua Lawrence and Haluai, Piyush and Tan, Mai and Crozier, Peter and Fernandez-Granda, Carlos}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {23937--23957}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/marcos-morales23a/marcos-morales23a.pdf}, url = {https://proceedings.mlr.press/v202/marcos-morales23a.html}, abstract = {Unsupervised denoising is a crucial challenge in real-world imaging applications. Unsupervised deep-learning methods have demonstrated impressive performance on benchmarks based on synthetic noise. However, no metrics exist to evaluate these methods in an unsupervised fashion. This is highly problematic for the many practical applications where ground-truth clean images are not available. In this work, we propose two novel metrics: the unsupervised mean squared error (MSE) and the unsupervised peak signal-to-noise ratio (PSNR), which are computed using only noisy data. We provide a theoretical analysis of these metrics, showing that they are asymptotically consistent estimators of the supervised MSE and PSNR. Controlled numerical experiments with synthetic noise confirm that they provide accurate approximations in practice. We validate our approach on real-world data from two imaging modalities: videos in raw format and transmission electron microscopy. Our results demonstrate that the proposed metrics enable unsupervised evaluation of denoising methods based exclusively on noisy data.} }
Endnote
%0 Conference Paper %T Evaluating Unsupervised Denoising Requires Unsupervised Metrics %A Adria Marcos Morales %A Matan Leibovich %A Sreyas Mohan %A Joshua Lawrence Vincent %A Piyush Haluai %A Mai Tan %A Peter Crozier %A Carlos Fernandez-Granda %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-marcos-morales23a %I PMLR %P 23937--23957 %U https://proceedings.mlr.press/v202/marcos-morales23a.html %V 202 %X Unsupervised denoising is a crucial challenge in real-world imaging applications. Unsupervised deep-learning methods have demonstrated impressive performance on benchmarks based on synthetic noise. However, no metrics exist to evaluate these methods in an unsupervised fashion. This is highly problematic for the many practical applications where ground-truth clean images are not available. In this work, we propose two novel metrics: the unsupervised mean squared error (MSE) and the unsupervised peak signal-to-noise ratio (PSNR), which are computed using only noisy data. We provide a theoretical analysis of these metrics, showing that they are asymptotically consistent estimators of the supervised MSE and PSNR. Controlled numerical experiments with synthetic noise confirm that they provide accurate approximations in practice. We validate our approach on real-world data from two imaging modalities: videos in raw format and transmission electron microscopy. Our results demonstrate that the proposed metrics enable unsupervised evaluation of denoising methods based exclusively on noisy data.
APA
Marcos Morales, A., Leibovich, M., Mohan, S., Vincent, J.L., Haluai, P., Tan, M., Crozier, P. & Fernandez-Granda, C.. (2023). Evaluating Unsupervised Denoising Requires Unsupervised Metrics. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:23937-23957 Available from https://proceedings.mlr.press/v202/marcos-morales23a.html.

Related Material