CancerGUIDE: Cancer Guideline Understanding via Internal Disagreement Estimation

Alyssa Unell, Noel C. F. Codella, J. Samuel Preston, Peniel Argaw, Wen-wai Yim, Zelalem Gero, Cliff Wong, Rajesh Jena, Eric Horvitz, Amanda K. Hall, Rachel Ruican Zhong, Jiachen Li, Shrey Jain, Mu Wei, Matthew P. Lungren, Hoifung Poon
Proceedings of the Fifth Machine Learning for Health Symposium, PMLR 297:275-294, 2026.

Abstract

The National Comprehensive Cancer Network ({NCCN}) provides evidence-based guidelines for cancer treatment. Translating complex patient presentations into guideline-compliant treatment recommendations is time-intensive, requires specialized expertise, and is prone to error. Advances in large language model ({LLM}) capabilities promise to reduce the time required to generate treatment recommendations and improve accuracy. We present an {LLM} agent-based approach to automatically generate guideline-concordant treatment trajectories for patients with non-small cell lung cancer ({NSCLC}). Our contributions are threefold. First, we construct a novel longitudinal dataset of 121 cases of {NSCLC} patients that includes clinical encounters, diagnostic results, and medical histories, each expertly annotated with the corresponding {NCCN} guideline trajectories by board-certified oncologists. Second, we demonstrate that existing {LLM}s possess domain-specific knowledge that enables high-quality proxy benchmark generation for both model development and evaluation, achieving strong correlation (Spearman coefficient r = 0.88, {RMSE} = 0.08) with expert-annotated benchmarks. Third, we develop a hybrid approach combining expensive human annotations with model consistency information to create both the agent framework that predicts the relevant guidelines for a patient, as well as a meta-classifier that verifies prediction accuracy with calibrated confidence scores for treatment recommendations ({AUROC} = 0.800). Calibrated confidence scoring is a critical capability for communicating the accuracy of outputs, custom-tailoring tradeoffs in performance, and supporting regulatory compliance. This work establishes a framework for clinically viable {LLM}-based guideline adherence systems that balance accuracy, interpretability, and regulatory requirements while reducing annotation costs, providing a scalable pathway toward automated clinical decision support.

Cite this Paper

BibTeX
@InProceedings{pmlr-v297-unell26a, title = {{CancerGUIDE}: Cancer Guideline Understanding via Internal Disagreement Estimation}, author = {Unell, Alyssa and Codella, Noel C. F. and Preston, J. Samuel and Argaw, Peniel and Yim, Wen-wai and Gero, Zelalem and Wong, Cliff and Jena, Rajesh and Horvitz, Eric and Hall, Amanda K. and Zhong, Rachel Ruican and Li, Jiachen and Jain, Shrey and Wei, Mu and Lungren, Matthew P. and Poon, Hoifung}, booktitle = {Proceedings of the Fifth Machine Learning for Health Symposium}, pages = {275--294}, year = {2026}, editor = {Argaw, Peniel and Zhang, Haoran and Jabbour, Sarah and Chandak, Payal and Ji, Jerry and Mukherjee, Sumit and Salaudeen, Olawale and Chang, Trenton and Healey, Elizabeth and Gröger, Fabian and Adibi, Amin and Hegselmann, Stefan and Wild, Benjamin and Noori, Ayush}, volume = {297}, series = {Proceedings of Machine Learning Research}, month = {13--14 Dec}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v297/main/assets/unell26a/unell26a.pdf}, url = {https://proceedings.mlr.press/v297/unell26a.html}, abstract = {The National Comprehensive Cancer Network ({NCCN}) provides evidence-based guidelines for cancer treatment. Translating complex patient presentations into guideline-compliant treatment recommendations is time-intensive, requires specialized expertise, and is prone to error. Advances in large language model ({LLM}) capabilities promise to reduce the time required to generate treatment recommendations and improve accuracy. We present an {LLM} agent-based approach to automatically generate guideline-concordant treatment trajectories for patients with non-small cell lung cancer ({NSCLC}). Our contributions are threefold. First, we construct a novel longitudinal dataset of 121 cases of {NSCLC} patients that includes clinical encounters, diagnostic results, and medical histories, each expertly annotated with the corresponding {NCCN} guideline trajectories by board-certified oncologists. Second, we demonstrate that existing {LLM}s possess domain-specific knowledge that enables high-quality proxy benchmark generation for both model development and evaluation, achieving strong correlation (Spearman coefficient r = 0.88, {RMSE} = 0.08) with expert-annotated benchmarks. Third, we develop a hybrid approach combining expensive human annotations with model consistency information to create both the agent framework that predicts the relevant guidelines for a patient, as well as a meta-classifier that verifies prediction accuracy with calibrated confidence scores for treatment recommendations ({AUROC} = 0.800). Calibrated confidence scoring is a critical capability for communicating the accuracy of outputs, custom-tailoring tradeoffs in performance, and supporting regulatory compliance. This work establishes a framework for clinically viable {LLM}-based guideline adherence systems that balance accuracy, interpretability, and regulatory requirements while reducing annotation costs, providing a scalable pathway toward automated clinical decision support.} }
Endnote
%0 Conference Paper %T CancerGUIDE: Cancer Guideline Understanding via Internal Disagreement Estimation %A Alyssa Unell %A Noel C. F. Codella %A J. Samuel Preston %A Peniel Argaw %A Wen-wai Yim %A Zelalem Gero %A Cliff Wong %A Rajesh Jena %A Eric Horvitz %A Amanda K. Hall %A Rachel Ruican Zhong %A Jiachen Li %A Shrey Jain %A Mu Wei %A Matthew P. Lungren %A Hoifung Poon %B Proceedings of the Fifth Machine Learning for Health Symposium %C Proceedings of Machine Learning Research %D 2026 %E Peniel Argaw %E Haoran Zhang %E Sarah Jabbour %E Payal Chandak %E Jerry Ji %E Sumit Mukherjee %E Olawale Salaudeen %E Trenton Chang %E Elizabeth Healey %E Fabian Gröger %E Amin Adibi %E Stefan Hegselmann %E Benjamin Wild %E Ayush Noori %F pmlr-v297-unell26a %I PMLR %P 275--294 %U https://proceedings.mlr.press/v297/unell26a.html %V 297 %X The National Comprehensive Cancer Network ({NCCN}) provides evidence-based guidelines for cancer treatment. Translating complex patient presentations into guideline-compliant treatment recommendations is time-intensive, requires specialized expertise, and is prone to error. Advances in large language model ({LLM}) capabilities promise to reduce the time required to generate treatment recommendations and improve accuracy. We present an {LLM} agent-based approach to automatically generate guideline-concordant treatment trajectories for patients with non-small cell lung cancer ({NSCLC}). Our contributions are threefold. First, we construct a novel longitudinal dataset of 121 cases of {NSCLC} patients that includes clinical encounters, diagnostic results, and medical histories, each expertly annotated with the corresponding {NCCN} guideline trajectories by board-certified oncologists. Second, we demonstrate that existing {LLM}s possess domain-specific knowledge that enables high-quality proxy benchmark generation for both model development and evaluation, achieving strong correlation (Spearman coefficient r = 0.88, {RMSE} = 0.08) with expert-annotated benchmarks. Third, we develop a hybrid approach combining expensive human annotations with model consistency information to create both the agent framework that predicts the relevant guidelines for a patient, as well as a meta-classifier that verifies prediction accuracy with calibrated confidence scores for treatment recommendations ({AUROC} = 0.800). Calibrated confidence scoring is a critical capability for communicating the accuracy of outputs, custom-tailoring tradeoffs in performance, and supporting regulatory compliance. This work establishes a framework for clinically viable {LLM}-based guideline adherence systems that balance accuracy, interpretability, and regulatory requirements while reducing annotation costs, providing a scalable pathway toward automated clinical decision support.
APA
Unell, A., Codella, N.C.F., Preston, J.S., Argaw, P., Yim, W., Gero, Z., Wong, C., Jena, R., Horvitz, E., Hall, A.K., Zhong, R.R., Li, J., Jain, S., Wei, M., Lungren, M.P. & Poon, H.. (2026). CancerGUIDE: Cancer Guideline Understanding via Internal Disagreement Estimation. Proceedings of the Fifth Machine Learning for Health Symposium, in Proceedings of Machine Learning Research 297:275-294 Available from https://proceedings.mlr.press/v297/unell26a.html.

Related Material