HeadCT-ONE: Enabling Granular and Controllable Automated Evaluation of Head CT Radiology Report Generation

Julian Nicolas Acosta, Xiaoman Zhang, Siddhant Dogra, Hong-Yu Zhou, Seyedmehdi Payabvash, Guido J. Falcone, Eric Karl Oermann, Pranav Rajpurkar
Proceedings of the sixth Conference on Health, Inference, and Learning, PMLR 287:649-671, 2025.

Abstract

We present Head CT Ontology Normalized Evaluation (HeadCT-ONE), a metric for evaluating head CT report generation through ontology-normalized entity and relation extraction. HeadCT-ONE enhances current information extraction derived metrics (such as RadGraph F1) by implementing entity normalization through domain-specific ontologies, addressing radiological language variability. HeadCT-ONE compares normalized entities and relations, allowing for controllable weighting of different entity types or specific entities. Through experiments on head CT reports from three health systems, we show that HeadCT-ONE’s normalization and weighting approach improves the capture of semantically equivalent reports, better distinguishes between normal and abnormal reports, and aligns with radiologists’ assessment of clinically significant errors, while offering flexibility to prioritize specific aspects of report content. Our results demonstrate how HeadCT-ONE enables more flexible, controllable, and granular automated evaluation of head CT reports.

Cite this Paper

BibTeX
@InProceedings{pmlr-v287-acosta25a, title = {HeadCT-ONE: Enabling Granular and Controllable Automated Evaluation of Head CT Radiology Report Generation}, author = {Acosta, Julian Nicolas and Zhang, Xiaoman and Dogra, Siddhant and Zhou, Hong-Yu and Payabvash, Seyedmehdi and Falcone, Guido J. and Oermann, Eric Karl and Rajpurkar, Pranav}, booktitle = {Proceedings of the sixth Conference on Health, Inference, and Learning}, pages = {649--671}, year = {2025}, editor = {Xu, Xuhai Orson and Choi, Edward and Singhal, Pankhuri and Gerych, Walter and Tang, Shengpu and Agrawal, Monica and Subbaswamy, Adarsh and Sizikova, Elena and Dunn, Jessilyn and Daneshjou, Roxana and Sarker, Tasmie and McDermott, Matthew and Chen, Irene}, volume = {287}, series = {Proceedings of Machine Learning Research}, month = {25--27 Jun}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v287/main/assets/acosta25a/acosta25a.pdf}, url = {https://proceedings.mlr.press/v287/acosta25a.html}, abstract = {We present Head CT Ontology Normalized Evaluation (HeadCT-ONE), a metric for evaluating head CT report generation through ontology-normalized entity and relation extraction. HeadCT-ONE enhances current information extraction derived metrics (such as RadGraph F1) by implementing entity normalization through domain-specific ontologies, addressing radiological language variability. HeadCT-ONE compares normalized entities and relations, allowing for controllable weighting of different entity types or specific entities. Through experiments on head CT reports from three health systems, we show that HeadCT-ONE’s normalization and weighting approach improves the capture of semantically equivalent reports, better distinguishes between normal and abnormal reports, and aligns with radiologists’ assessment of clinically significant errors, while offering flexibility to prioritize specific aspects of report content. Our results demonstrate how HeadCT-ONE enables more flexible, controllable, and granular automated evaluation of head CT reports.} }
Endnote
%0 Conference Paper %T HeadCT-ONE: Enabling Granular and Controllable Automated Evaluation of Head CT Radiology Report Generation %A Julian Nicolas Acosta %A Xiaoman Zhang %A Siddhant Dogra %A Hong-Yu Zhou %A Seyedmehdi Payabvash %A Guido J. Falcone %A Eric Karl Oermann %A Pranav Rajpurkar %B Proceedings of the sixth Conference on Health, Inference, and Learning %C Proceedings of Machine Learning Research %D 2025 %E Xuhai Orson Xu %E Edward Choi %E Pankhuri Singhal %E Walter Gerych %E Shengpu Tang %E Monica Agrawal %E Adarsh Subbaswamy %E Elena Sizikova %E Jessilyn Dunn %E Roxana Daneshjou %E Tasmie Sarker %E Matthew McDermott %E Irene Chen %F pmlr-v287-acosta25a %I PMLR %P 649--671 %U https://proceedings.mlr.press/v287/acosta25a.html %V 287 %X We present Head CT Ontology Normalized Evaluation (HeadCT-ONE), a metric for evaluating head CT report generation through ontology-normalized entity and relation extraction. HeadCT-ONE enhances current information extraction derived metrics (such as RadGraph F1) by implementing entity normalization through domain-specific ontologies, addressing radiological language variability. HeadCT-ONE compares normalized entities and relations, allowing for controllable weighting of different entity types or specific entities. Through experiments on head CT reports from three health systems, we show that HeadCT-ONE’s normalization and weighting approach improves the capture of semantically equivalent reports, better distinguishes between normal and abnormal reports, and aligns with radiologists’ assessment of clinically significant errors, while offering flexibility to prioritize specific aspects of report content. Our results demonstrate how HeadCT-ONE enables more flexible, controllable, and granular automated evaluation of head CT reports.
APA
Acosta, J.N., Zhang, X., Dogra, S., Zhou, H., Payabvash, S., Falcone, G.J., Oermann, E.K. & Rajpurkar, P.. (2025). HeadCT-ONE: Enabling Granular and Controllable Automated Evaluation of Head CT Radiology Report Generation. Proceedings of the sixth Conference on Health, Inference, and Learning, in Proceedings of Machine Learning Research 287:649-671 Available from https://proceedings.mlr.press/v287/acosta25a.html.

Related Material