Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priors

Vignav Ramesh, Nathan A. Chi, Pranav Rajpurkar
Proceedings of the 2nd Machine Learning for Health symposium, PMLR 193:456-473, 2022.

Abstract

Current deep learning models trained to generate radiology reports from chest radiographs are capable of producing clinically accurate, clear, and actionable text that can advance patient care. However, such systems all succumb to the same problem: making hallucinated references to non-existent prior reports. Such hallucinations occur because these models are trained on datasets of real-world patient reports that inherently refer to priors. To this end, we propose two methods to remove references to priors in radiology reports: (1) a GPT-3-based few-shot approach to rewrite medical reports without references to priors; and (2) a BioBERT-based token classification approach to directly remove words referring to priors. We use the aforementioned approaches to modify MIMIC-CXR, a publicly available dataset of chest X-rays and their associated free-text radiology reports; we then retrain CXR-RePaiR, a radiology report generation system, on the adapted MIMIC-CXR dataset. We find that our re-trained model—which we call CXR-ReDonE—outperforms previous report generation methods on clinical metrics, achieving an average BERTScore of 0.2351 ($2.57%$ absolute improvement). We expect our approach to be broadly valuable in enabling current radiology report generation systems to be more directly integrated into clinical pipelines.

Cite this Paper

BibTeX
@InProceedings{pmlr-v193-ramesh22a, title = {Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priors}, author = {Ramesh, Vignav and Chi, Nathan A. and Rajpurkar, Pranav}, booktitle = {Proceedings of the 2nd Machine Learning for Health symposium}, pages = {456--473}, year = {2022}, editor = {Parziale, Antonio and Agrawal, Monica and Joshi, Shalmali and Chen, Irene Y. and Tang, Shengpu and Oala, Luis and Subbaswamy, Adarsh}, volume = {193}, series = {Proceedings of Machine Learning Research}, month = {28 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v193/ramesh22a/ramesh22a.pdf}, url = {https://proceedings.mlr.press/v193/ramesh22a.html}, abstract = {Current deep learning models trained to generate radiology reports from chest radiographs are capable of producing clinically accurate, clear, and actionable text that can advance patient care. However, such systems all succumb to the same problem: making hallucinated references to non-existent prior reports. Such hallucinations occur because these models are trained on datasets of real-world patient reports that inherently refer to priors. To this end, we propose two methods to remove references to priors in radiology reports: (1) a GPT-3-based few-shot approach to rewrite medical reports without references to priors; and (2) a BioBERT-based token classification approach to directly remove words referring to priors. We use the aforementioned approaches to modify MIMIC-CXR, a publicly available dataset of chest X-rays and their associated free-text radiology reports; we then retrain CXR-RePaiR, a radiology report generation system, on the adapted MIMIC-CXR dataset. We find that our re-trained model—which we call CXR-ReDonE—outperforms previous report generation methods on clinical metrics, achieving an average BERTScore of 0.2351 ($2.57%$ absolute improvement). We expect our approach to be broadly valuable in enabling current radiology report generation systems to be more directly integrated into clinical pipelines. } }
Endnote
%0 Conference Paper %T Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priors %A Vignav Ramesh %A Nathan A. Chi %A Pranav Rajpurkar %B Proceedings of the 2nd Machine Learning for Health symposium %C Proceedings of Machine Learning Research %D 2022 %E Antonio Parziale %E Monica Agrawal %E Shalmali Joshi %E Irene Y. Chen %E Shengpu Tang %E Luis Oala %E Adarsh Subbaswamy %F pmlr-v193-ramesh22a %I PMLR %P 456--473 %U https://proceedings.mlr.press/v193/ramesh22a.html %V 193 %X Current deep learning models trained to generate radiology reports from chest radiographs are capable of producing clinically accurate, clear, and actionable text that can advance patient care. However, such systems all succumb to the same problem: making hallucinated references to non-existent prior reports. Such hallucinations occur because these models are trained on datasets of real-world patient reports that inherently refer to priors. To this end, we propose two methods to remove references to priors in radiology reports: (1) a GPT-3-based few-shot approach to rewrite medical reports without references to priors; and (2) a BioBERT-based token classification approach to directly remove words referring to priors. We use the aforementioned approaches to modify MIMIC-CXR, a publicly available dataset of chest X-rays and their associated free-text radiology reports; we then retrain CXR-RePaiR, a radiology report generation system, on the adapted MIMIC-CXR dataset. We find that our re-trained model—which we call CXR-ReDonE—outperforms previous report generation methods on clinical metrics, achieving an average BERTScore of 0.2351 ($2.57%$ absolute improvement). We expect our approach to be broadly valuable in enabling current radiology report generation systems to be more directly integrated into clinical pipelines.
APA
Ramesh, V., Chi, N.A. & Rajpurkar, P.. (2022). Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priors. Proceedings of the 2nd Machine Learning for Health symposium, in Proceedings of Machine Learning Research 193:456-473 Available from https://proceedings.mlr.press/v193/ramesh22a.html.

Related Material