Unseen Disease Detection for Deep Learning Interpretation of Chest X-rays

Siyu Shi, Ishaan Malhi, Kevin Tran, Andrew Y. Ng, Pranav Rajpurkar
Proceedings of the Fourth Conference on Medical Imaging with Deep Learning, PMLR 143:699-712, 2021.

Abstract

We systematically evaluate the performance of deep learning models in the presence of diseases not labeled for or present during training. First, we evaluate whether deep learning models trained on a subset of diseases (seen diseases) can detect the presence of any one of a larger set of diseases. We find that models tend to falsely classify diseases outside of the subset (unseen diseases) as ‘no disease”. Second, we evaluate whether models trained on seen diseases can detect seen diseases when co-occurring with diseases outside the subset (unseen diseases). We find that models are still able to detect seen diseases even when co-occurring with unseen diseases. Third, we evaluate whether feature representations learned by models may be used to detect the presence of unseen diseases given a small labeled set of unseen diseases. We find that the penultimate layer provides useful features for unseen disease detection. Our results can inform the safe clinical deployment of deep learning models trained on a non-exhaustive set of disease classes.

Cite this Paper

BibTeX
@InProceedings{pmlr-v143-shi21a, title = {Unseen Disease Detection for Deep Learning Interpretation of Chest X-rays}, author = {Shi, Siyu and Malhi, Ishaan and Tran, Kevin and Ng, Andrew Y. and Rajpurkar, Pranav}, booktitle = {Proceedings of the Fourth Conference on Medical Imaging with Deep Learning}, pages = {699--712}, year = {2021}, editor = {Heinrich, Mattias and Dou, Qi and de Bruijne, Marleen and Lellmann, Jan and Schläfer, Alexander and Ernst, Floris}, volume = {143}, series = {Proceedings of Machine Learning Research}, month = {07--09 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v143/shi21a/shi21a.pdf}, url = {https://proceedings.mlr.press/v143/shi21a.html}, abstract = {We systematically evaluate the performance of deep learning models in the presence of diseases not labeled for or present during training. First, we evaluate whether deep learning models trained on a subset of diseases (seen diseases) can detect the presence of any one of a larger set of diseases. We find that models tend to falsely classify diseases outside of the subset (unseen diseases) as ‘no disease”. Second, we evaluate whether models trained on seen diseases can detect seen diseases when co-occurring with diseases outside the subset (unseen diseases). We find that models are still able to detect seen diseases even when co-occurring with unseen diseases. Third, we evaluate whether feature representations learned by models may be used to detect the presence of unseen diseases given a small labeled set of unseen diseases. We find that the penultimate layer provides useful features for unseen disease detection. Our results can inform the safe clinical deployment of deep learning models trained on a non-exhaustive set of disease classes.} }
Endnote
%0 Conference Paper %T Unseen Disease Detection for Deep Learning Interpretation of Chest X-rays %A Siyu Shi %A Ishaan Malhi %A Kevin Tran %A Andrew Y. Ng %A Pranav Rajpurkar %B Proceedings of the Fourth Conference on Medical Imaging with Deep Learning %C Proceedings of Machine Learning Research %D 2021 %E Mattias Heinrich %E Qi Dou %E Marleen de Bruijne %E Jan Lellmann %E Alexander Schläfer %E Floris Ernst %F pmlr-v143-shi21a %I PMLR %P 699--712 %U https://proceedings.mlr.press/v143/shi21a.html %V 143 %X We systematically evaluate the performance of deep learning models in the presence of diseases not labeled for or present during training. First, we evaluate whether deep learning models trained on a subset of diseases (seen diseases) can detect the presence of any one of a larger set of diseases. We find that models tend to falsely classify diseases outside of the subset (unseen diseases) as ‘no disease”. Second, we evaluate whether models trained on seen diseases can detect seen diseases when co-occurring with diseases outside the subset (unseen diseases). We find that models are still able to detect seen diseases even when co-occurring with unseen diseases. Third, we evaluate whether feature representations learned by models may be used to detect the presence of unseen diseases given a small labeled set of unseen diseases. We find that the penultimate layer provides useful features for unseen disease detection. Our results can inform the safe clinical deployment of deep learning models trained on a non-exhaustive set of disease classes.
APA
Shi, S., Malhi, I., Tran, K., Ng, A.Y. & Rajpurkar, P.. (2021). Unseen Disease Detection for Deep Learning Interpretation of Chest X-rays. Proceedings of the Fourth Conference on Medical Imaging with Deep Learning, in Proceedings of Machine Learning Research 143:699-712 Available from https://proceedings.mlr.press/v143/shi21a.html.

Related Material