Moment Multicalibration for Uncertainty Estimation

Christopher Jung, Changhwa Lee, Mallesh Pai, Aaron Roth, Rakesh Vohra
Proceedings of Thirty Fourth Conference on Learning Theory, PMLR 134:2634-2678, 2021.

Abstract

We show how to achieve the notion of "multicalibration" from Hebert-Johnson et al. (2018) not just for means, but also for variances and other higher moments. Informally, this means that we can find regression functions which, given a data point, can make point predictions not just for the expectation of its label, but for higher moments of its label distribution as well—and those predictions match the true distribution quantities when averaged not just over the population as a whole, but also when averaged over an enormous number of finely defined subgroups. It yields a principled way to estimate the uncertainty of predictions on many different subgroups—and to diagnose potential sources of unfairness in the predictive power of features across subgroups. As an application, we show that our moment estimates can be used to derive marginal prediction intervals that are simultaneously valid as averaged over all of the (sufficiently large) subgroups for which moment multicalibration has been obtained.

Cite this Paper

BibTeX
@InProceedings{pmlr-v134-jung21a, title = {Moment Multicalibration for Uncertainty Estimation}, author = {Jung, Christopher and Lee, Changhwa and Pai, Mallesh and Roth, Aaron and Vohra, Rakesh}, booktitle = {Proceedings of Thirty Fourth Conference on Learning Theory}, pages = {2634--2678}, year = {2021}, editor = {Belkin, Mikhail and Kpotufe, Samory}, volume = {134}, series = {Proceedings of Machine Learning Research}, month = {15--19 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v134/jung21a/jung21a.pdf}, url = {https://proceedings.mlr.press/v134/jung21a.html}, abstract = {We show how to achieve the notion of "multicalibration" from Hebert-Johnson et al. (2018) not just for means, but also for variances and other higher moments. Informally, this means that we can find regression functions which, given a data point, can make point predictions not just for the expectation of its label, but for higher moments of its label distribution as well—and those predictions match the true distribution quantities when averaged not just over the population as a whole, but also when averaged over an enormous number of finely defined subgroups. It yields a principled way to estimate the uncertainty of predictions on many different subgroups—and to diagnose potential sources of unfairness in the predictive power of features across subgroups. As an application, we show that our moment estimates can be used to derive marginal prediction intervals that are simultaneously valid as averaged over all of the (sufficiently large) subgroups for which moment multicalibration has been obtained.} }
Endnote
%0 Conference Paper %T Moment Multicalibration for Uncertainty Estimation %A Christopher Jung %A Changhwa Lee %A Mallesh Pai %A Aaron Roth %A Rakesh Vohra %B Proceedings of Thirty Fourth Conference on Learning Theory %C Proceedings of Machine Learning Research %D 2021 %E Mikhail Belkin %E Samory Kpotufe %F pmlr-v134-jung21a %I PMLR %P 2634--2678 %U https://proceedings.mlr.press/v134/jung21a.html %V 134 %X We show how to achieve the notion of "multicalibration" from Hebert-Johnson et al. (2018) not just for means, but also for variances and other higher moments. Informally, this means that we can find regression functions which, given a data point, can make point predictions not just for the expectation of its label, but for higher moments of its label distribution as well—and those predictions match the true distribution quantities when averaged not just over the population as a whole, but also when averaged over an enormous number of finely defined subgroups. It yields a principled way to estimate the uncertainty of predictions on many different subgroups—and to diagnose potential sources of unfairness in the predictive power of features across subgroups. As an application, we show that our moment estimates can be used to derive marginal prediction intervals that are simultaneously valid as averaged over all of the (sufficiently large) subgroups for which moment multicalibration has been obtained.
APA
Jung, C., Lee, C., Pai, M., Roth, A. & Vohra, R.. (2021). Moment Multicalibration for Uncertainty Estimation. Proceedings of Thirty Fourth Conference on Learning Theory, in Proceedings of Machine Learning Research 134:2634-2678 Available from https://proceedings.mlr.press/v134/jung21a.html.

Related Material