Retrieval Augmented Chest X-Ray Report Generation using OpenAI GPT models

Mercy Ranjit, Gopinath Ganapathy, Ranjit Manuel, Tanuja Ganu
Proceedings of the 8th Machine Learning for Healthcare Conference, PMLR 219:650-666, 2023.

Abstract

We propose Retrieval Augmented Generation (RAG) as an approach for automated radiology report writing, using multimodally-aligned embeddings from a contrastively-pretrained vision language model to retrieve relevant radiology text for a given image, and then using a general domain generative model, such as OpenAI text-davinci-003, gpt-3.5-turbo and gpt-4, to generate a report based on the retrieved text. This approach keeps hallucinated generations under check and provides capabilities to generate report content in the format we desire, leveraging the instruction following capabilities of generative models. Our approach achieves better clinical metrics with a BERTScore of 0.2865 ($\Delta$+ 25.88 %) and Semb score of 0.4026 ($\Delta$+ 6.31 %). Our approach can be useful for different clinical settings, as it can augment the automated radiology report generation process with relevant content. It also allows to inject the user intents and requirements in the prompts, which can modulate the content and format of the generated reports according to the clinical setting.

Cite this Paper

BibTeX
@InProceedings{pmlr-v219-ranjit23a, title = {Retrieval Augmented Chest X-Ray Report Generation using OpenAI GPT models}, author = {Ranjit, Mercy and Ganapathy, Gopinath and Manuel, Ranjit and Ganu, Tanuja}, booktitle = {Proceedings of the 8th Machine Learning for Healthcare Conference}, pages = {650--666}, year = {2023}, editor = {Deshpande, Kaivalya and Fiterau, Madalina and Joshi, Shalmali and Lipton, Zachary and Ranganath, Rajesh and Urteaga, Iñigo and Yeung, Serene}, volume = {219}, series = {Proceedings of Machine Learning Research}, month = {11--12 Aug}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v219/ranjit23a/ranjit23a.pdf}, url = {https://proceedings.mlr.press/v219/ranjit23a.html}, abstract = {We propose Retrieval Augmented Generation (RAG) as an approach for automated radiology report writing, using multimodally-aligned embeddings from a contrastively-pretrained vision language model to retrieve relevant radiology text for a given image, and then using a general domain generative model, such as OpenAI text-davinci-003, gpt-3.5-turbo and gpt-4, to generate a report based on the retrieved text. This approach keeps hallucinated generations under check and provides capabilities to generate report content in the format we desire, leveraging the instruction following capabilities of generative models. Our approach achieves better clinical metrics with a BERTScore of 0.2865 ($\Delta$+ 25.88 %) and Semb score of 0.4026 ($\Delta$+ 6.31 %). Our approach can be useful for different clinical settings, as it can augment the automated radiology report generation process with relevant content. It also allows to inject the user intents and requirements in the prompts, which can modulate the content and format of the generated reports according to the clinical setting.} }
Endnote
%0 Conference Paper %T Retrieval Augmented Chest X-Ray Report Generation using OpenAI GPT models %A Mercy Ranjit %A Gopinath Ganapathy %A Ranjit Manuel %A Tanuja Ganu %B Proceedings of the 8th Machine Learning for Healthcare Conference %C Proceedings of Machine Learning Research %D 2023 %E Kaivalya Deshpande %E Madalina Fiterau %E Shalmali Joshi %E Zachary Lipton %E Rajesh Ranganath %E Iñigo Urteaga %E Serene Yeung %F pmlr-v219-ranjit23a %I PMLR %P 650--666 %U https://proceedings.mlr.press/v219/ranjit23a.html %V 219 %X We propose Retrieval Augmented Generation (RAG) as an approach for automated radiology report writing, using multimodally-aligned embeddings from a contrastively-pretrained vision language model to retrieve relevant radiology text for a given image, and then using a general domain generative model, such as OpenAI text-davinci-003, gpt-3.5-turbo and gpt-4, to generate a report based on the retrieved text. This approach keeps hallucinated generations under check and provides capabilities to generate report content in the format we desire, leveraging the instruction following capabilities of generative models. Our approach achieves better clinical metrics with a BERTScore of 0.2865 ($\Delta$+ 25.88 %) and Semb score of 0.4026 ($\Delta$+ 6.31 %). Our approach can be useful for different clinical settings, as it can augment the automated radiology report generation process with relevant content. It also allows to inject the user intents and requirements in the prompts, which can modulate the content and format of the generated reports according to the clinical setting.
APA
Ranjit, M., Ganapathy, G., Manuel, R. & Ganu, T.. (2023). Retrieval Augmented Chest X-Ray Report Generation using OpenAI GPT models. Proceedings of the 8th Machine Learning for Healthcare Conference, in Proceedings of Machine Learning Research 219:650-666 Available from https://proceedings.mlr.press/v219/ranjit23a.html.

Related Material