A Bayesian Approach to Quantify the Uncertainty of Human Ratings in a Single-Instance Multimodal Framework

Zijian Chen, Archana Venkataraman
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:17024-17038, 2026.

Abstract

Human ratings are central to learning and inference across several application domains, but they are also subject to inter-rater biases and judgment errors. Quantifying the uncertainty of these human ratings would require repeated measurements, which are expensive and rarely available at scale. We propose a Bayesian graphical model to estimate the instance-level and item-level uncertainty of (subjective) human ratings by leveraging auxiliary (objective) data. Our model learns a shared latent content representation that explains factors common to both the human rating and auxiliary data and a latent uncertainty variable that captures fluctuations in the human assessments via a data-conditioned prior. We develop a scalable amortized variational inference procedure that uses modality-appropriate neural encoders and decoders to represent the posterior factors. Experiments on synthetic data demonstrate that our framework can accurately recover the latent uncertainty under targeted ablations and stress tests. We further demonstrate our approach on a real-world dataset of paired functional MRI scans and behavioral testing for autism, thus highlighting the need for uncertainty quantification.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-chen26eh, title = {A {B}ayesian Approach to Quantify the Uncertainty of Human Ratings in a Single-Instance Multimodal Framework}, author = {Chen, Zijian and Venkataraman, Archana}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {17024--17038}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/chen26eh/chen26eh.pdf}, url = {https://proceedings.mlr.press/v306/chen26eh.html}, abstract = {Human ratings are central to learning and inference across several application domains, but they are also subject to inter-rater biases and judgment errors. Quantifying the uncertainty of these human ratings would require repeated measurements, which are expensive and rarely available at scale. We propose a Bayesian graphical model to estimate the instance-level and item-level uncertainty of (subjective) human ratings by leveraging auxiliary (objective) data. Our model learns a shared latent content representation that explains factors common to both the human rating and auxiliary data and a latent uncertainty variable that captures fluctuations in the human assessments via a data-conditioned prior. We develop a scalable amortized variational inference procedure that uses modality-appropriate neural encoders and decoders to represent the posterior factors. Experiments on synthetic data demonstrate that our framework can accurately recover the latent uncertainty under targeted ablations and stress tests. We further demonstrate our approach on a real-world dataset of paired functional MRI scans and behavioral testing for autism, thus highlighting the need for uncertainty quantification.} }
Endnote
%0 Conference Paper %T A Bayesian Approach to Quantify the Uncertainty of Human Ratings in a Single-Instance Multimodal Framework %A Zijian Chen %A Archana Venkataraman %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-chen26eh %I PMLR %P 17024--17038 %U https://proceedings.mlr.press/v306/chen26eh.html %V 306 %X Human ratings are central to learning and inference across several application domains, but they are also subject to inter-rater biases and judgment errors. Quantifying the uncertainty of these human ratings would require repeated measurements, which are expensive and rarely available at scale. We propose a Bayesian graphical model to estimate the instance-level and item-level uncertainty of (subjective) human ratings by leveraging auxiliary (objective) data. Our model learns a shared latent content representation that explains factors common to both the human rating and auxiliary data and a latent uncertainty variable that captures fluctuations in the human assessments via a data-conditioned prior. We develop a scalable amortized variational inference procedure that uses modality-appropriate neural encoders and decoders to represent the posterior factors. Experiments on synthetic data demonstrate that our framework can accurately recover the latent uncertainty under targeted ablations and stress tests. We further demonstrate our approach on a real-world dataset of paired functional MRI scans and behavioral testing for autism, thus highlighting the need for uncertainty quantification.
APA
Chen, Z. & Venkataraman, A.. (2026). A Bayesian Approach to Quantify the Uncertainty of Human Ratings in a Single-Instance Multimodal Framework. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:17024-17038 Available from https://proceedings.mlr.press/v306/chen26eh.html.

Related Material