Position: Benchmarks Cannot Establish Deployment Readiness of Clinical AI

Haoran Zhang, Hyewon Jeong, Olawale Elijah Salaudeen, Walter Gerych, Nigam Shah, Marzyeh Ghassemi
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:172915-172930, 2026.

Abstract

Despite large language models (LLMs) achieving impressive performance on benchmark tasks such as medical question answering, their real-world utility remains limited. We argue that while benchmarks play a valuable role in developing methods and filtering promising models during development, they fundamentally cannot establish deployment readiness. Many models topping benchmark performance have failed to perform seemingly related clinical tasks effectively in practice, while others with modest benchmark performance have demonstrated meaningful clinical benefits. We detail the limitations of benchmark-centric evaluations of deployment readiness. We argue that we should use benchmarks only to identify candidate methods or models, not to justify deployment. We call for increased use of other measurement instruments of deployment readiness, such as prospective studies, and policy changes that align incentives with clinically grounded evaluation.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-zhang26jq, title = {Position: Benchmarks Cannot Establish Deployment Readiness of Clinical {AI}}, author = {Zhang, Haoran and Jeong, Hyewon and Salaudeen, Olawale Elijah and Gerych, Walter and Shah, Nigam and Ghassemi, Marzyeh}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {172915--172930}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zhang26jq/zhang26jq.pdf}, url = {https://proceedings.mlr.press/v306/zhang26jq.html}, abstract = {Despite large language models (LLMs) achieving impressive performance on benchmark tasks such as medical question answering, their real-world utility remains limited. We argue that while benchmarks play a valuable role in developing methods and filtering promising models during development, they fundamentally cannot establish deployment readiness. Many models topping benchmark performance have failed to perform seemingly related clinical tasks effectively in practice, while others with modest benchmark performance have demonstrated meaningful clinical benefits. We detail the limitations of benchmark-centric evaluations of deployment readiness. We argue that we should use benchmarks only to identify candidate methods or models, not to justify deployment. We call for increased use of other measurement instruments of deployment readiness, such as prospective studies, and policy changes that align incentives with clinically grounded evaluation.} }
Endnote
%0 Conference Paper %T Position: Benchmarks Cannot Establish Deployment Readiness of Clinical AI %A Haoran Zhang %A Hyewon Jeong %A Olawale Elijah Salaudeen %A Walter Gerych %A Nigam Shah %A Marzyeh Ghassemi %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zhang26jq %I PMLR %P 172915--172930 %U https://proceedings.mlr.press/v306/zhang26jq.html %V 306 %X Despite large language models (LLMs) achieving impressive performance on benchmark tasks such as medical question answering, their real-world utility remains limited. We argue that while benchmarks play a valuable role in developing methods and filtering promising models during development, they fundamentally cannot establish deployment readiness. Many models topping benchmark performance have failed to perform seemingly related clinical tasks effectively in practice, while others with modest benchmark performance have demonstrated meaningful clinical benefits. We detail the limitations of benchmark-centric evaluations of deployment readiness. We argue that we should use benchmarks only to identify candidate methods or models, not to justify deployment. We call for increased use of other measurement instruments of deployment readiness, such as prospective studies, and policy changes that align incentives with clinically grounded evaluation.
APA
Zhang, H., Jeong, H., Salaudeen, O.E., Gerych, W., Shah, N. & Ghassemi, M.. (2026). Position: Benchmarks Cannot Establish Deployment Readiness of Clinical AI. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:172915-172930 Available from https://proceedings.mlr.press/v306/zhang26jq.html.

Related Material