Predictive Complexity Priors

Eric Nalisnick, Jonathan Gordon, Jose Miguel Hernandez-Lobato
Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, PMLR 130:694-702, 2021.

Abstract

Specifying a Bayesian prior is notoriously difficult for complex models such as neural networks. Reasoning about parameters is made challenging by the high-dimensionality and over-parameterization of the space. Priors that seem benign and uninformative can have unintuitive and detrimental effects on a model’s predictions. For this reason, we propose predictive complexity priors: a functional prior that is defined by comparing the model’s predictions to those of a reference model. Although originally defined on the model outputs, we transfer the prior to the model parameters via a change of variables. The traditional Bayesian workflow can then proceed as usual. We apply our predictive complexity prior to high-dimensional regression, reasoning over neural network depth, and sharing of statistical strength for few-shot learning.

Cite this Paper

BibTeX
@InProceedings{pmlr-v130-nalisnick21a, title = { Predictive Complexity Priors }, author = {Nalisnick, Eric and Gordon, Jonathan and Miguel Hernandez-Lobato, Jose}, booktitle = {Proceedings of The 24th International Conference on Artificial Intelligence and Statistics}, pages = {694--702}, year = {2021}, editor = {Banerjee, Arindam and Fukumizu, Kenji}, volume = {130}, series = {Proceedings of Machine Learning Research}, month = {13--15 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v130/nalisnick21a/nalisnick21a.pdf}, url = {https://proceedings.mlr.press/v130/nalisnick21a.html}, abstract = { Specifying a Bayesian prior is notoriously difficult for complex models such as neural networks. Reasoning about parameters is made challenging by the high-dimensionality and over-parameterization of the space. Priors that seem benign and uninformative can have unintuitive and detrimental effects on a model’s predictions. For this reason, we propose predictive complexity priors: a functional prior that is defined by comparing the model’s predictions to those of a reference model. Although originally defined on the model outputs, we transfer the prior to the model parameters via a change of variables. The traditional Bayesian workflow can then proceed as usual. We apply our predictive complexity prior to high-dimensional regression, reasoning over neural network depth, and sharing of statistical strength for few-shot learning. } }
Endnote
%0 Conference Paper %T Predictive Complexity Priors %A Eric Nalisnick %A Jonathan Gordon %A Jose Miguel Hernandez-Lobato %B Proceedings of The 24th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2021 %E Arindam Banerjee %E Kenji Fukumizu %F pmlr-v130-nalisnick21a %I PMLR %P 694--702 %U https://proceedings.mlr.press/v130/nalisnick21a.html %V 130 %X Specifying a Bayesian prior is notoriously difficult for complex models such as neural networks. Reasoning about parameters is made challenging by the high-dimensionality and over-parameterization of the space. Priors that seem benign and uninformative can have unintuitive and detrimental effects on a model’s predictions. For this reason, we propose predictive complexity priors: a functional prior that is defined by comparing the model’s predictions to those of a reference model. Although originally defined on the model outputs, we transfer the prior to the model parameters via a change of variables. The traditional Bayesian workflow can then proceed as usual. We apply our predictive complexity prior to high-dimensional regression, reasoning over neural network depth, and sharing of statistical strength for few-shot learning.
APA
Nalisnick, E., Gordon, J. & Miguel Hernandez-Lobato, J.. (2021). Predictive Complexity Priors . Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 130:694-702 Available from https://proceedings.mlr.press/v130/nalisnick21a.html.

Related Material