QuEst: Enhancing Estimates of Quantile-Based Distributional Measures Using Model Predictions

Zhun Deng, Thomas P Zollo, Benjamin Eyre, Amogh Inamdar, David Madras, Richard Zemel
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:13347-13368, 2025.

Abstract

As machine learning models grow increasingly competent, their predictions can supplement scarce or expensive data in various important domains. In support of this paradigm, algorithms have emerged to combine a small amount of high-fidelity observed data with a much larger set of imputed model outputs to estimate some quantity of interest. Yet current hybrid-inference tools target only means or single quantiles, limiting their applicability for many critical domains and use cases. We present QuEst, a principled framework to merge observed and imputed data to deliver point estimates and rigorous confidence intervals for a wide family of quantile-based distributional measures. QuEst covers a range of measures, from tail risk (CVaR) to population segments such as quartiles, that are central to fields such as economics, sociology, education, medicine, and more. We extend QuEst to multidimensional metrics, and introduce an additional optimization technique to further reduce variance in this and other hybrid estimators. We demonstrate the utility of our framework through experiments in economic modeling, opinion polling, and language model auto-evaluation.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-deng25k, title = {{Q}u{E}st: Enhancing Estimates of Quantile-Based Distributional Measures Using Model Predictions}, author = {Deng, Zhun and Zollo, Thomas P and Eyre, Benjamin and Inamdar, Amogh and Madras, David and Zemel, Richard}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {13347--13368}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/deng25k/deng25k.pdf}, url = {https://proceedings.mlr.press/v267/deng25k.html}, abstract = {As machine learning models grow increasingly competent, their predictions can supplement scarce or expensive data in various important domains. In support of this paradigm, algorithms have emerged to combine a small amount of high-fidelity observed data with a much larger set of imputed model outputs to estimate some quantity of interest. Yet current hybrid-inference tools target only means or single quantiles, limiting their applicability for many critical domains and use cases. We present QuEst, a principled framework to merge observed and imputed data to deliver point estimates and rigorous confidence intervals for a wide family of quantile-based distributional measures. QuEst covers a range of measures, from tail risk (CVaR) to population segments such as quartiles, that are central to fields such as economics, sociology, education, medicine, and more. We extend QuEst to multidimensional metrics, and introduce an additional optimization technique to further reduce variance in this and other hybrid estimators. We demonstrate the utility of our framework through experiments in economic modeling, opinion polling, and language model auto-evaluation.} }
Endnote
%0 Conference Paper %T QuEst: Enhancing Estimates of Quantile-Based Distributional Measures Using Model Predictions %A Zhun Deng %A Thomas P Zollo %A Benjamin Eyre %A Amogh Inamdar %A David Madras %A Richard Zemel %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-deng25k %I PMLR %P 13347--13368 %U https://proceedings.mlr.press/v267/deng25k.html %V 267 %X As machine learning models grow increasingly competent, their predictions can supplement scarce or expensive data in various important domains. In support of this paradigm, algorithms have emerged to combine a small amount of high-fidelity observed data with a much larger set of imputed model outputs to estimate some quantity of interest. Yet current hybrid-inference tools target only means or single quantiles, limiting their applicability for many critical domains and use cases. We present QuEst, a principled framework to merge observed and imputed data to deliver point estimates and rigorous confidence intervals for a wide family of quantile-based distributional measures. QuEst covers a range of measures, from tail risk (CVaR) to population segments such as quartiles, that are central to fields such as economics, sociology, education, medicine, and more. We extend QuEst to multidimensional metrics, and introduce an additional optimization technique to further reduce variance in this and other hybrid estimators. We demonstrate the utility of our framework through experiments in economic modeling, opinion polling, and language model auto-evaluation.
APA
Deng, Z., Zollo, T.P., Eyre, B., Inamdar, A., Madras, D. & Zemel, R.. (2025). QuEst: Enhancing Estimates of Quantile-Based Distributional Measures Using Model Predictions. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:13347-13368 Available from https://proceedings.mlr.press/v267/deng25k.html.

Related Material