CLINIC : Evaluating Multilingual Trustworthiness in Language Models for Healthcare

Akash Ghosh, Srivarshinee Sridhar, Raghav Kaushik Ravi, Muhsin Muhsin, Sriparna Saha, Chirag Agarwal
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:34790-34840, 2026.

Abstract

Integrating language models (LMs) in healthcare systems holds great promise for improving medical workflows and decision-making. However, a critical barrier to their global adoption is the lack of reliable evaluation of their trustworthiness in multilingual healthcare settings. Existing LMs are predominantly trained in high-resource languages, making them ill-equipped to handle the complexity and diversity of healthcare queries in mid- and low-resource languages, which poses significant challenges for deployment in global healthcare contexts where linguistic diversity is essential. In this work, we present CLINIC, a Comprehensive Multilingual Benchmark to evaluate the trustworthiness of language models in healthcare. CLINIC systematically benchmarks LMs across five key dimensions of trustworthiness: truthfulness, fairness, safety, robustness, and privacy, operationalized through 18 diverse tasks spanning 15 languages and covering a wide range of critical healthcare topics. Our extensive evaluation reveals that LMs struggle with factual correctness, demonstrate bias across demographic and linguistic groups, and remain susceptible to privacy breaches and adversarial attacks. By highlighting these shortcomings, CLINIC lays the foundation for enhancing the global reach and safety of LMs in healthcare across diverse languages.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-ghosh26b, title = {{CLINIC} : Evaluating Multilingual Trustworthiness in Language Models for Healthcare}, author = {Ghosh, Akash and Sridhar, Srivarshinee and Ravi, Raghav Kaushik and Muhsin, Muhsin and Saha, Sriparna and Agarwal, Chirag}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {34790--34840}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/ghosh26b/ghosh26b.pdf}, url = {https://proceedings.mlr.press/v306/ghosh26b.html}, abstract = {Integrating language models (LMs) in healthcare systems holds great promise for improving medical workflows and decision-making. However, a critical barrier to their global adoption is the lack of reliable evaluation of their trustworthiness in multilingual healthcare settings. Existing LMs are predominantly trained in high-resource languages, making them ill-equipped to handle the complexity and diversity of healthcare queries in mid- and low-resource languages, which poses significant challenges for deployment in global healthcare contexts where linguistic diversity is essential. In this work, we present CLINIC, a Comprehensive Multilingual Benchmark to evaluate the trustworthiness of language models in healthcare. CLINIC systematically benchmarks LMs across five key dimensions of trustworthiness: truthfulness, fairness, safety, robustness, and privacy, operationalized through 18 diverse tasks spanning 15 languages and covering a wide range of critical healthcare topics. Our extensive evaluation reveals that LMs struggle with factual correctness, demonstrate bias across demographic and linguistic groups, and remain susceptible to privacy breaches and adversarial attacks. By highlighting these shortcomings, CLINIC lays the foundation for enhancing the global reach and safety of LMs in healthcare across diverse languages.} }
Endnote
%0 Conference Paper %T CLINIC : Evaluating Multilingual Trustworthiness in Language Models for Healthcare %A Akash Ghosh %A Srivarshinee Sridhar %A Raghav Kaushik Ravi %A Muhsin Muhsin %A Sriparna Saha %A Chirag Agarwal %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-ghosh26b %I PMLR %P 34790--34840 %U https://proceedings.mlr.press/v306/ghosh26b.html %V 306 %X Integrating language models (LMs) in healthcare systems holds great promise for improving medical workflows and decision-making. However, a critical barrier to their global adoption is the lack of reliable evaluation of their trustworthiness in multilingual healthcare settings. Existing LMs are predominantly trained in high-resource languages, making them ill-equipped to handle the complexity and diversity of healthcare queries in mid- and low-resource languages, which poses significant challenges for deployment in global healthcare contexts where linguistic diversity is essential. In this work, we present CLINIC, a Comprehensive Multilingual Benchmark to evaluate the trustworthiness of language models in healthcare. CLINIC systematically benchmarks LMs across five key dimensions of trustworthiness: truthfulness, fairness, safety, robustness, and privacy, operationalized through 18 diverse tasks spanning 15 languages and covering a wide range of critical healthcare topics. Our extensive evaluation reveals that LMs struggle with factual correctness, demonstrate bias across demographic and linguistic groups, and remain susceptible to privacy breaches and adversarial attacks. By highlighting these shortcomings, CLINIC lays the foundation for enhancing the global reach and safety of LMs in healthcare across diverse languages.
APA
Ghosh, A., Sridhar, S., Ravi, R.K., Muhsin, M., Saha, S. & Agarwal, C.. (2026). CLINIC : Evaluating Multilingual Trustworthiness in Language Models for Healthcare. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:34790-34840 Available from https://proceedings.mlr.press/v306/ghosh26b.html.

Related Material