Learning Generalized Medical Image Representations Through Image-Graph Contrastive Pretraining

Sameer Khanna, Daniel Michael, Marinka Zitnik, Pranav Rajpurkar
Proceedings of the 3rd Machine Learning for Health Symposium, PMLR 225:232-243, 2023.

Abstract

Medical image interpretation using deep learning has shown promise but often requires extensive expert-annotated datasets. To reduce this annotation burden, we develop an Image-Graph Contrastive Learning framework that pairs chest X-rays with structured report knowledge graphs automatically extracted from radiology notes. Our approach uniquely encodes the disconnected graph components via a relational graph convolution network and transformer attention. In experiments on the CheXpert dataset, this novel graph encoding strategy enabled the framework to outperform existing methods that use image-text contrastive learning in 1 {\}% linear evaluation and few-shot settings, while achieving comparable performance to radiologists. By exploiting unlabeled paired images and text, our framework demonstrates the potential of structured clinical insights to enhance contrastive learning for medical images. This work points toward reducing demands on medical experts for annotations, improving diagnostic precision, and advancing patient care through robust medical image understanding.

Cite this Paper

BibTeX
@InProceedings{pmlr-v225-khanna23a, title = {Learning Generalized Medical Image Representations Through Image-Graph Contrastive Pretraining}, author = {Khanna, Sameer and Michael, Daniel and Zitnik, Marinka and Rajpurkar, Pranav}, booktitle = {Proceedings of the 3rd Machine Learning for Health Symposium}, pages = {232--243}, year = {2023}, editor = {Hegselmann, Stefan and Parziale, Antonio and Shanmugam, Divya and Tang, Shengpu and Asiedu, Mercy Nyamewaa and Chang, Serina and Hartvigsen, Tom and Singh, Harvineet}, volume = {225}, series = {Proceedings of Machine Learning Research}, month = {10 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v225/khanna23a/khanna23a.pdf}, url = {https://proceedings.mlr.press/v225/khanna23a.html}, abstract = {Medical image interpretation using deep learning has shown promise but often requires extensive expert-annotated datasets. To reduce this annotation burden, we develop an Image-Graph Contrastive Learning framework that pairs chest X-rays with structured report knowledge graphs automatically extracted from radiology notes. Our approach uniquely encodes the disconnected graph components via a relational graph convolution network and transformer attention. In experiments on the CheXpert dataset, this novel graph encoding strategy enabled the framework to outperform existing methods that use image-text contrastive learning in 1 {\}% linear evaluation and few-shot settings, while achieving comparable performance to radiologists. By exploiting unlabeled paired images and text, our framework demonstrates the potential of structured clinical insights to enhance contrastive learning for medical images. This work points toward reducing demands on medical experts for annotations, improving diagnostic precision, and advancing patient care through robust medical image understanding.} }
Endnote
%0 Conference Paper %T Learning Generalized Medical Image Representations Through Image-Graph Contrastive Pretraining %A Sameer Khanna %A Daniel Michael %A Marinka Zitnik %A Pranav Rajpurkar %B Proceedings of the 3rd Machine Learning for Health Symposium %C Proceedings of Machine Learning Research %D 2023 %E Stefan Hegselmann %E Antonio Parziale %E Divya Shanmugam %E Shengpu Tang %E Mercy Nyamewaa Asiedu %E Serina Chang %E Tom Hartvigsen %E Harvineet Singh %F pmlr-v225-khanna23a %I PMLR %P 232--243 %U https://proceedings.mlr.press/v225/khanna23a.html %V 225 %X Medical image interpretation using deep learning has shown promise but often requires extensive expert-annotated datasets. To reduce this annotation burden, we develop an Image-Graph Contrastive Learning framework that pairs chest X-rays with structured report knowledge graphs automatically extracted from radiology notes. Our approach uniquely encodes the disconnected graph components via a relational graph convolution network and transformer attention. In experiments on the CheXpert dataset, this novel graph encoding strategy enabled the framework to outperform existing methods that use image-text contrastive learning in 1 {\}% linear evaluation and few-shot settings, while achieving comparable performance to radiologists. By exploiting unlabeled paired images and text, our framework demonstrates the potential of structured clinical insights to enhance contrastive learning for medical images. This work points toward reducing demands on medical experts for annotations, improving diagnostic precision, and advancing patient care through robust medical image understanding.
APA
Khanna, S., Michael, D., Zitnik, M. & Rajpurkar, P.. (2023). Learning Generalized Medical Image Representations Through Image-Graph Contrastive Pretraining. Proceedings of the 3rd Machine Learning for Health Symposium, in Proceedings of Machine Learning Research 225:232-243 Available from https://proceedings.mlr.press/v225/khanna23a.html.

Related Material