Retrieve, Then Classify: Corpus-Grounded Automation of Clinical Value Set Authoring

Sumit Mukherjee, O. Tate Kernell, Nairwita Mazumder, Juan Shu, Celena A. Wheeler, Chris Gibbons
Proceedings of the 11th Machine Learning for Healthcare Conference, PMLR 340:1321-1339, 2026.

Abstract

Clinical value set authoring — the task of identifying all codes in a standardized vocabulary that define a clinical concept — is a recurring bottleneck in clinical quality measurement and phenotyping. A natural approach is to prompt a large language model (LLM) to generate the required codes directly, but structured clinical vocabularies are large, version-controlled, and not reliably memorized during pretraining. We propose Retrieval-Augmented Set Completion (RASC): retrieve the $K$ most similar existing value sets from a curated corpus to form a candidate pool, then apply a classifier to each candidate code. Theoretically, retrieve-and-select can reduce statistical complexity by shrinking the effective output space from the full vocabulary to a much smaller retrieved candidate pool. We demonstrate the utility of RASC on 11,803 publicly available VSAC value sets, constructing the first large-scale benchmark for this task. A cross-encoder fine-tuned on SAPBert achieves AUROC 0.852 and value-set-level F1 0.298, outperforming a simpler three-layer Multilayer Perceptron (AUROC 0.799, F1 0.250) and both reduce the number of irrelevant candidates per true positive from 12.3 (retrieval-only) to approximately 3.2 and 4.4 respectively. Zero-shot GPT-4o achieves value-set-level F1 0.105, with 48.6% of returned codes absent from VSAC entirely. This performance gap widens with increasing value set size, consistent with RASC’s theoretical advantage. We observe similar performance gains across two other classifier model types, namely a cross-encoder initialized from pre-trained SAPBert and a LightGBM model, demonstrating that RASC’s benefits extend beyond a single model class.

Cite this Paper

BibTeX
@InProceedings{pmlr-v340-mukherjee26a, title = {Retrieve, Then Classify: Corpus-Grounded Automation of Clinical Value Set Authoring}, author = {Mukherjee, Sumit and Kernell, O. Tate and Mazumder, Nairwita and Shu, Juan and Wheeler, Celena A. and Gibbons, Chris}, booktitle = {Proceedings of the 11th Machine Learning for Healthcare Conference}, pages = {1321--1339}, year = {2026}, editor = {Krishnan, Rahul G. and van Amsterdam, Wouter A. C. and Chopra, Sumit and Overgaard, Shauna and Hughes, Michael and Ötleş, Erkin and Shen, Yiqiu and Shanmugam, Divya and Nayan, Madhur and Engelhard, Matthew and Fackler, Jim and Oberst, Michael}, volume = {340}, series = {Proceedings of Machine Learning Research}, month = {12--14 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v340/main/assets/mukherjee26a/mukherjee26a.pdf}, url = {https://proceedings.mlr.press/v340/mukherjee26a.html}, abstract = {Clinical value set authoring — the task of identifying all codes in a standardized vocabulary that define a clinical concept — is a recurring bottleneck in clinical quality measurement and phenotyping. A natural approach is to prompt a large language model (LLM) to generate the required codes directly, but structured clinical vocabularies are large, version-controlled, and not reliably memorized during pretraining. We propose Retrieval-Augmented Set Completion (RASC): retrieve the $K$ most similar existing value sets from a curated corpus to form a candidate pool, then apply a classifier to each candidate code. Theoretically, retrieve-and-select can reduce statistical complexity by shrinking the effective output space from the full vocabulary to a much smaller retrieved candidate pool. We demonstrate the utility of RASC on 11,803 publicly available VSAC value sets, constructing the first large-scale benchmark for this task. A cross-encoder fine-tuned on SAPBert achieves AUROC 0.852 and value-set-level F1 0.298, outperforming a simpler three-layer Multilayer Perceptron (AUROC 0.799, F1 0.250) and both reduce the number of irrelevant candidates per true positive from 12.3 (retrieval-only) to approximately 3.2 and 4.4 respectively. Zero-shot GPT-4o achieves value-set-level F1 0.105, with 48.6% of returned codes absent from VSAC entirely. This performance gap widens with increasing value set size, consistent with RASC’s theoretical advantage. We observe similar performance gains across two other classifier model types, namely a cross-encoder initialized from pre-trained SAPBert and a LightGBM model, demonstrating that RASC’s benefits extend beyond a single model class.} }
Endnote
%0 Conference Paper %T Retrieve, Then Classify: Corpus-Grounded Automation of Clinical Value Set Authoring %A Sumit Mukherjee %A O. Tate Kernell %A Nairwita Mazumder %A Juan Shu %A Celena A. Wheeler %A Chris Gibbons %B Proceedings of the 11th Machine Learning for Healthcare Conference %C Proceedings of Machine Learning Research %D 2026 %E Rahul G. Krishnan %E Wouter A. C. van Amsterdam %E Sumit Chopra %E Shauna Overgaard %E Michael Hughes %E Erkin Ötleş %E Yiqiu Shen %E Divya Shanmugam %E Madhur Nayan %E Matthew Engelhard %E Jim Fackler %E Michael Oberst %F pmlr-v340-mukherjee26a %I PMLR %P 1321--1339 %U https://proceedings.mlr.press/v340/mukherjee26a.html %V 340 %X Clinical value set authoring — the task of identifying all codes in a standardized vocabulary that define a clinical concept — is a recurring bottleneck in clinical quality measurement and phenotyping. A natural approach is to prompt a large language model (LLM) to generate the required codes directly, but structured clinical vocabularies are large, version-controlled, and not reliably memorized during pretraining. We propose Retrieval-Augmented Set Completion (RASC): retrieve the $K$ most similar existing value sets from a curated corpus to form a candidate pool, then apply a classifier to each candidate code. Theoretically, retrieve-and-select can reduce statistical complexity by shrinking the effective output space from the full vocabulary to a much smaller retrieved candidate pool. We demonstrate the utility of RASC on 11,803 publicly available VSAC value sets, constructing the first large-scale benchmark for this task. A cross-encoder fine-tuned on SAPBert achieves AUROC 0.852 and value-set-level F1 0.298, outperforming a simpler three-layer Multilayer Perceptron (AUROC 0.799, F1 0.250) and both reduce the number of irrelevant candidates per true positive from 12.3 (retrieval-only) to approximately 3.2 and 4.4 respectively. Zero-shot GPT-4o achieves value-set-level F1 0.105, with 48.6% of returned codes absent from VSAC entirely. This performance gap widens with increasing value set size, consistent with RASC’s theoretical advantage. We observe similar performance gains across two other classifier model types, namely a cross-encoder initialized from pre-trained SAPBert and a LightGBM model, demonstrating that RASC’s benefits extend beyond a single model class.
APA
Mukherjee, S., Kernell, O.T., Mazumder, N., Shu, J., Wheeler, C.A. & Gibbons, C.. (2026). Retrieve, Then Classify: Corpus-Grounded Automation of Clinical Value Set Authoring. Proceedings of the 11th Machine Learning for Healthcare Conference, in Proceedings of Machine Learning Research 340:1321-1339 Available from https://proceedings.mlr.press/v340/mukherjee26a.html.

Related Material