Retrieval-Based Chest X-Ray Report Generation Using a Pre-trained Contrastive Language-Image Model

Mark Endo, Rayan Krishnan, Viswesh Krishna, Andrew Y. Ng, Pranav Rajpurkar
Proceedings of Machine Learning for Health, PMLR 158:209-219, 2021.

Abstract

We propose CXR-RePaiR: a retrieval-based radiology report generation approach using a pre-trained contrastive language-image model. Our method generates clinically accurate reports on both in-distribution and out-of-distribution data. CXR-RePaiR outperforms or matches prior report generation methods on clinical metrics, achieving an average F$_1$ score of 0.352 ($\Delta$ + 7.98%) on an external radiology dataset (CheXpert). Further, we implement a compression approach used to reduce the size of the reference corpus and speed up the runtime of our retrieval method. With compression, our model maintains similar performance while producing reports 70% faster than the best generative model. Our approach can be broadly useful in improving the diagnostic performance and generalizability of report generation models and enabling their use in clinical workflows.

Cite this Paper

BibTeX
@InProceedings{pmlr-v158-endo21a, title = {Retrieval-Based Chest X-Ray Report Generation Using a Pre-trained Contrastive Language-Image Model}, author = {Endo, Mark and Krishnan, Rayan and Krishna, Viswesh and Ng, Andrew Y. and Rajpurkar, Pranav}, booktitle = {Proceedings of Machine Learning for Health}, pages = {209--219}, year = {2021}, editor = {Roy, Subhrajit and Pfohl, Stephen and Rocheteau, Emma and Tadesse, Girmaw Abebe and Oala, Luis and Falck, Fabian and Zhou, Yuyin and Shen, Liyue and Zamzmi, Ghada and Mugambi, Purity and Zirikly, Ayah and McDermott, Matthew B. A. and Alsentzer, Emily}, volume = {158}, series = {Proceedings of Machine Learning Research}, month = {04 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v158/endo21a/endo21a.pdf}, url = {https://proceedings.mlr.press/v158/endo21a.html}, abstract = {We propose CXR-RePaiR: a retrieval-based radiology report generation approach using a pre-trained contrastive language-image model. Our method generates clinically accurate reports on both in-distribution and out-of-distribution data. CXR-RePaiR outperforms or matches prior report generation methods on clinical metrics, achieving an average F$_1$ score of 0.352 ($\Delta$ + 7.98%) on an external radiology dataset (CheXpert). Further, we implement a compression approach used to reduce the size of the reference corpus and speed up the runtime of our retrieval method. With compression, our model maintains similar performance while producing reports 70% faster than the best generative model. Our approach can be broadly useful in improving the diagnostic performance and generalizability of report generation models and enabling their use in clinical workflows.} }
Endnote
%0 Conference Paper %T Retrieval-Based Chest X-Ray Report Generation Using a Pre-trained Contrastive Language-Image Model %A Mark Endo %A Rayan Krishnan %A Viswesh Krishna %A Andrew Y. Ng %A Pranav Rajpurkar %B Proceedings of Machine Learning for Health %C Proceedings of Machine Learning Research %D 2021 %E Subhrajit Roy %E Stephen Pfohl %E Emma Rocheteau %E Girmaw Abebe Tadesse %E Luis Oala %E Fabian Falck %E Yuyin Zhou %E Liyue Shen %E Ghada Zamzmi %E Purity Mugambi %E Ayah Zirikly %E Matthew B. A. McDermott %E Emily Alsentzer %F pmlr-v158-endo21a %I PMLR %P 209--219 %U https://proceedings.mlr.press/v158/endo21a.html %V 158 %X We propose CXR-RePaiR: a retrieval-based radiology report generation approach using a pre-trained contrastive language-image model. Our method generates clinically accurate reports on both in-distribution and out-of-distribution data. CXR-RePaiR outperforms or matches prior report generation methods on clinical metrics, achieving an average F$_1$ score of 0.352 ($\Delta$ + 7.98%) on an external radiology dataset (CheXpert). Further, we implement a compression approach used to reduce the size of the reference corpus and speed up the runtime of our retrieval method. With compression, our model maintains similar performance while producing reports 70% faster than the best generative model. Our approach can be broadly useful in improving the diagnostic performance and generalizability of report generation models and enabling their use in clinical workflows.
APA
Endo, M., Krishnan, R., Krishna, V., Ng, A.Y. & Rajpurkar, P.. (2021). Retrieval-Based Chest X-Ray Report Generation Using a Pre-trained Contrastive Language-Image Model. Proceedings of Machine Learning for Health, in Proceedings of Machine Learning Research 158:209-219 Available from https://proceedings.mlr.press/v158/endo21a.html.

Related Material