Privacy Audits for Clinical Large Language Models

Florent Pollet, Kirill Nikitin, Tong Wang, Rahul Gupta, Noémie Elhadad, Gamze Gursoy
Proceedings of the 11th Machine Learning for Healthcare Conference, PMLR 340:1517-1536, 2026.

Abstract

Large language models (LLMs) fine-tuned on de-identified clinical notes raise privacy concerns because automated de-identification can leave residual patient identifiers in the training data. We study whether such identifiers can be recovered from a fine-tuned model using query access alone as a function of query budget. We introduce Verified Extraction, an auditing framework that distinguishes identifiers attributable to fine-tuning data from spurious or prior-driven outputs and quantifies recoverable leakage under explicit query budgets. Using MIMIC-IV-Note as the fine-tuning dataset, we find that verified leakage is negligible at small query budgets but becomes practically significant under repeated querying, even when only a small fraction of identifiers remains in the training data. These results highlight the importance of privacy evaluations that account for repeated-query access rather than one-off prompt tests.

Cite this Paper

BibTeX
@InProceedings{pmlr-v340-pollet26a, title = {Privacy Audits for Clinical Large Language Models}, author = {Pollet, Florent and Nikitin, Kirill and Wang, Tong and Gupta, Rahul and Elhadad, No\'{e}mie and Gursoy, Gamze}, booktitle = {Proceedings of the 11th Machine Learning for Healthcare Conference}, pages = {1517--1536}, year = {2026}, editor = {Krishnan, Rahul G. and van Amsterdam, Wouter A. C. and Chopra, Sumit and Overgaard, Shauna and Hughes, Michael and Ötleş, Erkin and Shen, Yiqiu and Shanmugam, Divya and Nayan, Madhur and Engelhard, Matthew and Fackler, Jim and Oberst, Michael}, volume = {340}, series = {Proceedings of Machine Learning Research}, month = {12--14 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v340/main/assets/pollet26a/pollet26a.pdf}, url = {https://proceedings.mlr.press/v340/pollet26a.html}, abstract = {Large language models (LLMs) fine-tuned on de-identified clinical notes raise privacy concerns because automated de-identification can leave residual patient identifiers in the training data. We study whether such identifiers can be recovered from a fine-tuned model using query access alone as a function of query budget. We introduce Verified Extraction, an auditing framework that distinguishes identifiers attributable to fine-tuning data from spurious or prior-driven outputs and quantifies recoverable leakage under explicit query budgets. Using MIMIC-IV-Note as the fine-tuning dataset, we find that verified leakage is negligible at small query budgets but becomes practically significant under repeated querying, even when only a small fraction of identifiers remains in the training data. These results highlight the importance of privacy evaluations that account for repeated-query access rather than one-off prompt tests.} }
Endnote
%0 Conference Paper %T Privacy Audits for Clinical Large Language Models %A Florent Pollet %A Kirill Nikitin %A Tong Wang %A Rahul Gupta %A Noémie Elhadad %A Gamze Gursoy %B Proceedings of the 11th Machine Learning for Healthcare Conference %C Proceedings of Machine Learning Research %D 2026 %E Rahul G. Krishnan %E Wouter A. C. van Amsterdam %E Sumit Chopra %E Shauna Overgaard %E Michael Hughes %E Erkin Ötleş %E Yiqiu Shen %E Divya Shanmugam %E Madhur Nayan %E Matthew Engelhard %E Jim Fackler %E Michael Oberst %F pmlr-v340-pollet26a %I PMLR %P 1517--1536 %U https://proceedings.mlr.press/v340/pollet26a.html %V 340 %X Large language models (LLMs) fine-tuned on de-identified clinical notes raise privacy concerns because automated de-identification can leave residual patient identifiers in the training data. We study whether such identifiers can be recovered from a fine-tuned model using query access alone as a function of query budget. We introduce Verified Extraction, an auditing framework that distinguishes identifiers attributable to fine-tuning data from spurious or prior-driven outputs and quantifies recoverable leakage under explicit query budgets. Using MIMIC-IV-Note as the fine-tuning dataset, we find that verified leakage is negligible at small query budgets but becomes practically significant under repeated querying, even when only a small fraction of identifiers remains in the training data. These results highlight the importance of privacy evaluations that account for repeated-query access rather than one-off prompt tests.
APA
Pollet, F., Nikitin, K., Wang, T., Gupta, R., Elhadad, N. & Gursoy, G.. (2026). Privacy Audits for Clinical Large Language Models. Proceedings of the 11th Machine Learning for Healthcare Conference, in Proceedings of Machine Learning Research 340:1517-1536 Available from https://proceedings.mlr.press/v340/pollet26a.html.

Related Material