Bayesian Inference for Correlated Human Experts and Classifiers

Markelle Kelly, Alex James Boyd, Sam Showalter, Mark Steyvers, Padhraic Smyth
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:29670-29686, 2025.

Abstract

Applications of machine learning often involve making predictions based on both model outputs and the opinions of human experts. In this context, we investigate the problem of querying experts for class label predictions, using as few human queries as possible, and leveraging the class probability estimates of pre-trained classifiers. We develop a general Bayesian framework for this problem, modeling expert correlation via a joint latent representation, enabling simulation-based inference about the utility of additional expert queries, as well as inference of posterior distributions over unobserved expert labels. We apply our approach to two real-world medical classification problems, as well as to CIFAR-10H and ImageNet-16H, demonstrating substantial reductions relative to baselines in the cost of querying human experts while maintaining high prediction accuracy.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-kelly25a, title = {{B}ayesian Inference for Correlated Human Experts and Classifiers}, author = {Kelly, Markelle and Boyd, Alex James and Showalter, Sam and Steyvers, Mark and Smyth, Padhraic}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {29670--29686}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/kelly25a/kelly25a.pdf}, url = {https://proceedings.mlr.press/v267/kelly25a.html}, abstract = {Applications of machine learning often involve making predictions based on both model outputs and the opinions of human experts. In this context, we investigate the problem of querying experts for class label predictions, using as few human queries as possible, and leveraging the class probability estimates of pre-trained classifiers. We develop a general Bayesian framework for this problem, modeling expert correlation via a joint latent representation, enabling simulation-based inference about the utility of additional expert queries, as well as inference of posterior distributions over unobserved expert labels. We apply our approach to two real-world medical classification problems, as well as to CIFAR-10H and ImageNet-16H, demonstrating substantial reductions relative to baselines in the cost of querying human experts while maintaining high prediction accuracy.} }
Endnote
%0 Conference Paper %T Bayesian Inference for Correlated Human Experts and Classifiers %A Markelle Kelly %A Alex James Boyd %A Sam Showalter %A Mark Steyvers %A Padhraic Smyth %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-kelly25a %I PMLR %P 29670--29686 %U https://proceedings.mlr.press/v267/kelly25a.html %V 267 %X Applications of machine learning often involve making predictions based on both model outputs and the opinions of human experts. In this context, we investigate the problem of querying experts for class label predictions, using as few human queries as possible, and leveraging the class probability estimates of pre-trained classifiers. We develop a general Bayesian framework for this problem, modeling expert correlation via a joint latent representation, enabling simulation-based inference about the utility of additional expert queries, as well as inference of posterior distributions over unobserved expert labels. We apply our approach to two real-world medical classification problems, as well as to CIFAR-10H and ImageNet-16H, demonstrating substantial reductions relative to baselines in the cost of querying human experts while maintaining high prediction accuracy.
APA
Kelly, M., Boyd, A.J., Showalter, S., Steyvers, M. & Smyth, P.. (2025). Bayesian Inference for Correlated Human Experts and Classifiers. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:29670-29686 Available from https://proceedings.mlr.press/v267/kelly25a.html.

Related Material