Evaluating imprecise forecasts

Jason Konek
Proceedings of the Thirteenth International Symposium on Imprecise Probability: Theories and Applications, PMLR 215:270-279, 2023.

Abstract

This paper will introduce a new class of IP scoring rules for sets of almost desirable gambles. A set of almost desirable gambles $\mathcal{D}$ is evaluable for what might be called generalised type 1 and type 2 error. Generalised type 1 error is roughly a matter of the extent to which $\mathcal{D}$ encodes false judgments of desirability. Generalised type 2 error is roughly a matter of the extent to which $\mathcal{D}$ fails to encode true judgments of desirability. IP scoring rules are penalty functions that average these two types of error. To demonstrate the viability of IP scoring rules, we must show that for any coherent $\mathcal{D}$ you might choose, we can construct an IP scoring rule that renders it admissible. Moreover, every other admissible model relative to that scoring rule is also coherent. This paper makes progress toward that goal. We will also compare the class of scoring rules developed here with the results by Seidenfeld, Schervish, and Kadane from 2012,which establish that there is no strictly proper, continuous real-valued scoring rule for lower and upper probability forecasts.

Cite this Paper

BibTeX
@InProceedings{pmlr-v215-konek23a, title = {Evaluating imprecise forecasts}, author = {Konek, Jason}, booktitle = {Proceedings of the Thirteenth International Symposium on Imprecise Probability: Theories and Applications}, pages = {270--279}, year = {2023}, editor = {Miranda, Enrique and Montes, Ignacio and Quaeghebeur, Erik and Vantaggi, Barbara}, volume = {215}, series = {Proceedings of Machine Learning Research}, month = {11--14 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v215/konek23a/konek23a.pdf}, url = {https://proceedings.mlr.press/v215/konek23a.html}, abstract = {This paper will introduce a new class of IP scoring rules for sets of almost desirable gambles. A set of almost desirable gambles $\mathcal{D}$ is evaluable for what might be called generalised type 1 and type 2 error. Generalised type 1 error is roughly a matter of the extent to which $\mathcal{D}$ encodes false judgments of desirability. Generalised type 2 error is roughly a matter of the extent to which $\mathcal{D}$ fails to encode true judgments of desirability. IP scoring rules are penalty functions that average these two types of error. To demonstrate the viability of IP scoring rules, we must show that for any coherent $\mathcal{D}$ you might choose, we can construct an IP scoring rule that renders it admissible. Moreover, every other admissible model relative to that scoring rule is also coherent. This paper makes progress toward that goal. We will also compare the class of scoring rules developed here with the results by Seidenfeld, Schervish, and Kadane from 2012,which establish that there is no strictly proper, continuous real-valued scoring rule for lower and upper probability forecasts.} }
Endnote
%0 Conference Paper %T Evaluating imprecise forecasts %A Jason Konek %B Proceedings of the Thirteenth International Symposium on Imprecise Probability: Theories and Applications %C Proceedings of Machine Learning Research %D 2023 %E Enrique Miranda %E Ignacio Montes %E Erik Quaeghebeur %E Barbara Vantaggi %F pmlr-v215-konek23a %I PMLR %P 270--279 %U https://proceedings.mlr.press/v215/konek23a.html %V 215 %X This paper will introduce a new class of IP scoring rules for sets of almost desirable gambles. A set of almost desirable gambles $\mathcal{D}$ is evaluable for what might be called generalised type 1 and type 2 error. Generalised type 1 error is roughly a matter of the extent to which $\mathcal{D}$ encodes false judgments of desirability. Generalised type 2 error is roughly a matter of the extent to which $\mathcal{D}$ fails to encode true judgments of desirability. IP scoring rules are penalty functions that average these two types of error. To demonstrate the viability of IP scoring rules, we must show that for any coherent $\mathcal{D}$ you might choose, we can construct an IP scoring rule that renders it admissible. Moreover, every other admissible model relative to that scoring rule is also coherent. This paper makes progress toward that goal. We will also compare the class of scoring rules developed here with the results by Seidenfeld, Schervish, and Kadane from 2012,which establish that there is no strictly proper, continuous real-valued scoring rule for lower and upper probability forecasts.
APA
Konek, J.. (2023). Evaluating imprecise forecasts. Proceedings of the Thirteenth International Symposium on Imprecise Probability: Theories and Applications, in Proceedings of Machine Learning Research 215:270-279 Available from https://proceedings.mlr.press/v215/konek23a.html.

Related Material