An Analysis of the Impact of Annotation Errors on the Accuracy of Deep Learning for Cell Segmentation

Şerban Vădineanu, Daniël Maria Pelt, Oleh Dzyubachyk, Kees Joost Batenburg
Proceedings of The 5th International Conference on Medical Imaging with Deep Learning, PMLR 172:1251-1267, 2022.

Abstract

Recent studies have shown that there can be high inter- and intra-observer variability when creating annotations for biomedical image segmentation. To mitigate the effects of manual annotation variability when training machine learning algorithms, various methods have been developed. However, little work has been done on actually assessing the impact of annotation errors on machine learning-based segmentation. For the task of cell segmentation, our work aims to bridge this gap by providing a thorough analysis of three types of potential annotation errors. We tackle the limitation of previous studies that lack a golden standard ground truth by performing our analysis on two synthetically-generated data sets with perfect labels, while also validating our observations on manually-labeled data. Moreover, we discuss the influence of the annotation errors on the results of three different network architectures: UNet, SegNet, and MSD. We find that UNet shows the overall best robustness for all data sets on two categories of errors, especially when the severity of the error is low, while MSD generalizes well even when a large proportion of the cell labels is missing during training. Moreover, we observe that special care should be taken to avoid wrongly labeling large objects when the target cells have small footprints.

Cite this Paper

BibTeX
@InProceedings{pmlr-v172-vadineanu22a, title = {An Analysis of the Impact of Annotation Errors on the Accuracy of Deep Learning for Cell Segmentation}, author = {V\u{a}dineanu, \c{S}erban and Pelt, Dani{\"e}l Maria and Dzyubachyk, Oleh and Batenburg, Kees Joost}, booktitle = {Proceedings of The 5th International Conference on Medical Imaging with Deep Learning}, pages = {1251--1267}, year = {2022}, editor = {Konukoglu, Ender and Menze, Bjoern and Venkataraman, Archana and Baumgartner, Christian and Dou, Qi and Albarqouni, Shadi}, volume = {172}, series = {Proceedings of Machine Learning Research}, month = {06--08 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v172/vadineanu22a/vadineanu22a.pdf}, url = {https://proceedings.mlr.press/v172/vadineanu22a.html}, abstract = {Recent studies have shown that there can be high inter- and intra-observer variability when creating annotations for biomedical image segmentation. To mitigate the effects of manual annotation variability when training machine learning algorithms, various methods have been developed. However, little work has been done on actually assessing the impact of annotation errors on machine learning-based segmentation. For the task of cell segmentation, our work aims to bridge this gap by providing a thorough analysis of three types of potential annotation errors. We tackle the limitation of previous studies that lack a golden standard ground truth by performing our analysis on two synthetically-generated data sets with perfect labels, while also validating our observations on manually-labeled data. Moreover, we discuss the influence of the annotation errors on the results of three different network architectures: UNet, SegNet, and MSD. We find that UNet shows the overall best robustness for all data sets on two categories of errors, especially when the severity of the error is low, while MSD generalizes well even when a large proportion of the cell labels is missing during training. Moreover, we observe that special care should be taken to avoid wrongly labeling large objects when the target cells have small footprints.} }
Endnote
%0 Conference Paper %T An Analysis of the Impact of Annotation Errors on the Accuracy of Deep Learning for Cell Segmentation %A Şerban Vădineanu %A Daniël Maria Pelt %A Oleh Dzyubachyk %A Kees Joost Batenburg %B Proceedings of The 5th International Conference on Medical Imaging with Deep Learning %C Proceedings of Machine Learning Research %D 2022 %E Ender Konukoglu %E Bjoern Menze %E Archana Venkataraman %E Christian Baumgartner %E Qi Dou %E Shadi Albarqouni %F pmlr-v172-vadineanu22a %I PMLR %P 1251--1267 %U https://proceedings.mlr.press/v172/vadineanu22a.html %V 172 %X Recent studies have shown that there can be high inter- and intra-observer variability when creating annotations for biomedical image segmentation. To mitigate the effects of manual annotation variability when training machine learning algorithms, various methods have been developed. However, little work has been done on actually assessing the impact of annotation errors on machine learning-based segmentation. For the task of cell segmentation, our work aims to bridge this gap by providing a thorough analysis of three types of potential annotation errors. We tackle the limitation of previous studies that lack a golden standard ground truth by performing our analysis on two synthetically-generated data sets with perfect labels, while also validating our observations on manually-labeled data. Moreover, we discuss the influence of the annotation errors on the results of three different network architectures: UNet, SegNet, and MSD. We find that UNet shows the overall best robustness for all data sets on two categories of errors, especially when the severity of the error is low, while MSD generalizes well even when a large proportion of the cell labels is missing during training. Moreover, we observe that special care should be taken to avoid wrongly labeling large objects when the target cells have small footprints.
APA
Vădineanu, Ş., Pelt, D.M., Dzyubachyk, O. & Batenburg, K.J.. (2022). An Analysis of the Impact of Annotation Errors on the Accuracy of Deep Learning for Cell Segmentation. Proceedings of The 5th International Conference on Medical Imaging with Deep Learning, in Proceedings of Machine Learning Research 172:1251-1267 Available from https://proceedings.mlr.press/v172/vadineanu22a.html.

Related Material