Towards Explainable End-to-End Prostate Cancer Relapse Prediction from H&E Images Combining Self-Attention Multiple Instance Learning with a Recurrent Neural Network

Esther Dietrich, Patrick Fuhlert, Anne Ernst, Guido Sauter, Maximilian Lennartz, H. Siegfried Stiehl, Marina Zimmermann, Stefan Bonn
Proceedings of Machine Learning for Health, PMLR 158:38-53, 2021.

Abstract

Clinical decision support for histopathology image data mainly focuses on strongly supervised annotations, which offers intuitive interpretability, but is bound by expert performance. Here, we propose an explainable cancer relapse prediction network (eCaReNet) and show that end-to-end learning without strong annotations offers state-of-the-art performance while interpretability can be included through an attention mechanism. On the use case of prostate cancer survival prediction, using 14,479 images and only relapse times as annotations, we reach a cumulative dynamic AUC of 0.78 on a validation set, being on par with an expert pathologist (and an AUC of 0.77 on a separate test set). Our model is well-calibrated and outputs survival curves as well as a risk score and group per patient. Making use of the attention weights of a multiple instance learning layer, we show that malignant patches have a higher influence on the prediction than benign patches, thus offering an intuitive interpretation of the prediction. Our code is available at www.github.com/imsb-uke/ecarenet.

Cite this Paper

BibTeX
@InProceedings{pmlr-v158-dietrich21a, title = {Towards Explainable End-to-End Prostate Cancer Relapse Prediction from H&E Images Combining Self-Attention Multiple Instance Learning with a Recurrent Neural Network}, author = {Dietrich, Esther and Fuhlert, Patrick and Ernst, Anne and Sauter, Guido and Lennartz, Maximilian and Stiehl, H. Siegfried and Zimmermann, Marina and Bonn, Stefan}, booktitle = {Proceedings of Machine Learning for Health}, pages = {38--53}, year = {2021}, editor = {Roy, Subhrajit and Pfohl, Stephen and Rocheteau, Emma and Tadesse, Girmaw Abebe and Oala, Luis and Falck, Fabian and Zhou, Yuyin and Shen, Liyue and Zamzmi, Ghada and Mugambi, Purity and Zirikly, Ayah and McDermott, Matthew B. A. and Alsentzer, Emily}, volume = {158}, series = {Proceedings of Machine Learning Research}, month = {04 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v158/dietrich21a/dietrich21a.pdf}, url = {https://proceedings.mlr.press/v158/dietrich21a.html}, abstract = {Clinical decision support for histopathology image data mainly focuses on strongly supervised annotations, which offers intuitive interpretability, but is bound by expert performance. Here, we propose an explainable cancer relapse prediction network (eCaReNet) and show that end-to-end learning without strong annotations offers state-of-the-art performance while interpretability can be included through an attention mechanism. On the use case of prostate cancer survival prediction, using 14,479 images and only relapse times as annotations, we reach a cumulative dynamic AUC of 0.78 on a validation set, being on par with an expert pathologist (and an AUC of 0.77 on a separate test set). Our model is well-calibrated and outputs survival curves as well as a risk score and group per patient. Making use of the attention weights of a multiple instance learning layer, we show that malignant patches have a higher influence on the prediction than benign patches, thus offering an intuitive interpretation of the prediction. Our code is available at www.github.com/imsb-uke/ecarenet.} }
Endnote
%0 Conference Paper %T Towards Explainable End-to-End Prostate Cancer Relapse Prediction from H&E Images Combining Self-Attention Multiple Instance Learning with a Recurrent Neural Network %A Esther Dietrich %A Patrick Fuhlert %A Anne Ernst %A Guido Sauter %A Maximilian Lennartz %A H. Siegfried Stiehl %A Marina Zimmermann %A Stefan Bonn %B Proceedings of Machine Learning for Health %C Proceedings of Machine Learning Research %D 2021 %E Subhrajit Roy %E Stephen Pfohl %E Emma Rocheteau %E Girmaw Abebe Tadesse %E Luis Oala %E Fabian Falck %E Yuyin Zhou %E Liyue Shen %E Ghada Zamzmi %E Purity Mugambi %E Ayah Zirikly %E Matthew B. A. McDermott %E Emily Alsentzer %F pmlr-v158-dietrich21a %I PMLR %P 38--53 %U https://proceedings.mlr.press/v158/dietrich21a.html %V 158 %X Clinical decision support for histopathology image data mainly focuses on strongly supervised annotations, which offers intuitive interpretability, but is bound by expert performance. Here, we propose an explainable cancer relapse prediction network (eCaReNet) and show that end-to-end learning without strong annotations offers state-of-the-art performance while interpretability can be included through an attention mechanism. On the use case of prostate cancer survival prediction, using 14,479 images and only relapse times as annotations, we reach a cumulative dynamic AUC of 0.78 on a validation set, being on par with an expert pathologist (and an AUC of 0.77 on a separate test set). Our model is well-calibrated and outputs survival curves as well as a risk score and group per patient. Making use of the attention weights of a multiple instance learning layer, we show that malignant patches have a higher influence on the prediction than benign patches, thus offering an intuitive interpretation of the prediction. Our code is available at www.github.com/imsb-uke/ecarenet.
APA
Dietrich, E., Fuhlert, P., Ernst, A., Sauter, G., Lennartz, M., Stiehl, H.S., Zimmermann, M. & Bonn, S.. (2021). Towards Explainable End-to-End Prostate Cancer Relapse Prediction from H&E Images Combining Self-Attention Multiple Instance Learning with a Recurrent Neural Network. Proceedings of Machine Learning for Health, in Proceedings of Machine Learning Research 158:38-53 Available from https://proceedings.mlr.press/v158/dietrich21a.html.

Related Material