A Deployment Audit of Release-Side Risk in Conformal Triage under Prevalence Shift

Chengze Li, Xiao Liu, Hanrong Zhang, Haiyang Peng, Yanghao Ruan, Huanhuan Ma, Chunyu Miao, Qichao Zhou, Xiangrong Qi, Philip Yu
Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 329:793-812, 2026.

Abstract

Conformal triage converts predictive scores into deployment actions that either release a case, flag it for urgent attention, or defer it to human review. Under an observed change in target-event prevalence, however, marginal coverage and human-review rate can miss whether patients who experience the target event are released without review. To address this gap, we introduce a leakage-aware deployment audit for release-side conformal triage. It first assigns target subjects to three non-overlapping roles: prevalence correction, conformal calibration, and held-out release-side evaluation. This separation then lets the audit evaluate release directly: how many event-positive patients are cleared without review, whether the pilot has enough event labels for calibration, and how the release-review trade-off shifts. Applying this audit to a retrospective non-small-cell lung cancer (NSCLC) target cohort shows why lower review can be misleading: after prevalence correction, the pooled conformal branch lowers review by releasing more patients, some of whom are event-positive. Within the audit, the classwise branch acts as a scarcity diagnostic: the pilot has too few event labels to support a low-review release rule.

Cite this Paper

BibTeX
@InProceedings{pmlr-v329-li26a, title = {A Deployment Audit of Release-Side Risk in Conformal Triage under Prevalence Shift}, author = {Li, Chengze and Liu, Xiao and Zhang, Hanrong and Peng, Haiyang and Ruan, Yanghao and Ma, Huanhuan and Miao, Chunyu and Zhou, Qichao and Qi, Xiangrong and Yu, Philip}, booktitle = {Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications}, pages = {793--812}, year = {2026}, editor = {Ahlberg, Ernst and Johansson, Ulf and Boström, Henrik and Carlevaro, Alberto and Hallberg Szabadváry, Johan and Carlsson, Lars}, volume = {329}, series = {Proceedings of Machine Learning Research}, month = {02--04 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v329/main/assets/li26a/li26a.pdf}, url = {https://proceedings.mlr.press/v329/li26a.html}, abstract = {Conformal triage converts predictive scores into deployment actions that either release a case, flag it for urgent attention, or defer it to human review. Under an observed change in target-event prevalence, however, marginal coverage and human-review rate can miss whether patients who experience the target event are released without review. To address this gap, we introduce a leakage-aware deployment audit for release-side conformal triage. It first assigns target subjects to three non-overlapping roles: prevalence correction, conformal calibration, and held-out release-side evaluation. This separation then lets the audit evaluate release directly: how many event-positive patients are cleared without review, whether the pilot has enough event labels for calibration, and how the release-review trade-off shifts. Applying this audit to a retrospective non-small-cell lung cancer (NSCLC) target cohort shows why lower review can be misleading: after prevalence correction, the pooled conformal branch lowers review by releasing more patients, some of whom are event-positive. Within the audit, the classwise branch acts as a scarcity diagnostic: the pilot has too few event labels to support a low-review release rule.} }
Endnote
%0 Conference Paper %T A Deployment Audit of Release-Side Risk in Conformal Triage under Prevalence Shift %A Chengze Li %A Xiao Liu %A Hanrong Zhang %A Haiyang Peng %A Yanghao Ruan %A Huanhuan Ma %A Chunyu Miao %A Qichao Zhou %A Xiangrong Qi %A Philip Yu %B Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications %C Proceedings of Machine Learning Research %D 2026 %E Ernst Ahlberg %E Ulf Johansson %E Henrik Boström %E Alberto Carlevaro %E Johan Hallberg Szabadváry %E Lars Carlsson %F pmlr-v329-li26a %I PMLR %P 793--812 %U https://proceedings.mlr.press/v329/li26a.html %V 329 %X Conformal triage converts predictive scores into deployment actions that either release a case, flag it for urgent attention, or defer it to human review. Under an observed change in target-event prevalence, however, marginal coverage and human-review rate can miss whether patients who experience the target event are released without review. To address this gap, we introduce a leakage-aware deployment audit for release-side conformal triage. It first assigns target subjects to three non-overlapping roles: prevalence correction, conformal calibration, and held-out release-side evaluation. This separation then lets the audit evaluate release directly: how many event-positive patients are cleared without review, whether the pilot has enough event labels for calibration, and how the release-review trade-off shifts. Applying this audit to a retrospective non-small-cell lung cancer (NSCLC) target cohort shows why lower review can be misleading: after prevalence correction, the pooled conformal branch lowers review by releasing more patients, some of whom are event-positive. Within the audit, the classwise branch acts as a scarcity diagnostic: the pilot has too few event labels to support a low-review release rule.
APA
Li, C., Liu, X., Zhang, H., Peng, H., Ruan, Y., Ma, H., Miao, C., Zhou, Q., Qi, X. & Yu, P.. (2026). A Deployment Audit of Release-Side Risk in Conformal Triage under Prevalence Shift. Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications, in Proceedings of Machine Learning Research 329:793-812 Available from https://proceedings.mlr.press/v329/li26a.html.

Related Material