Learning Expressive Priors for Generalization and Uncertainty Estimation in Neural Networks

Dominik Schnaus, Jongseok Lee, Daniel Cremers, Rudolph Triebel
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:30252-30284, 2023.

Abstract

In this work, we propose a novel prior learning method for advancing generalization and uncertainty estimation in deep neural networks. The key idea is to exploit scalable and structured posteriors of neural networks as informative priors with generalization guarantees. Our learned priors provide expressive probabilistic representations at large scale, like Bayesian counterparts of pre-trained models on ImageNet, and further produce non-vacuous generalization bounds. We also extend this idea to a continual learning framework, where the favorable properties of our priors are desirable. Major enablers are our technical contributions: (1) the sums-of-Kronecker-product computations, and (2) the derivations and optimizations of tractable objectives that lead to improved generalization bounds. Empirically, we exhaustively show the effectiveness of this method for uncertainty estimation and generalization.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-schnaus23a, title = {Learning Expressive Priors for Generalization and Uncertainty Estimation in Neural Networks}, author = {Schnaus, Dominik and Lee, Jongseok and Cremers, Daniel and Triebel, Rudolph}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {30252--30284}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/schnaus23a/schnaus23a.pdf}, url = {https://proceedings.mlr.press/v202/schnaus23a.html}, abstract = {In this work, we propose a novel prior learning method for advancing generalization and uncertainty estimation in deep neural networks. The key idea is to exploit scalable and structured posteriors of neural networks as informative priors with generalization guarantees. Our learned priors provide expressive probabilistic representations at large scale, like Bayesian counterparts of pre-trained models on ImageNet, and further produce non-vacuous generalization bounds. We also extend this idea to a continual learning framework, where the favorable properties of our priors are desirable. Major enablers are our technical contributions: (1) the sums-of-Kronecker-product computations, and (2) the derivations and optimizations of tractable objectives that lead to improved generalization bounds. Empirically, we exhaustively show the effectiveness of this method for uncertainty estimation and generalization.} }
Endnote
%0 Conference Paper %T Learning Expressive Priors for Generalization and Uncertainty Estimation in Neural Networks %A Dominik Schnaus %A Jongseok Lee %A Daniel Cremers %A Rudolph Triebel %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-schnaus23a %I PMLR %P 30252--30284 %U https://proceedings.mlr.press/v202/schnaus23a.html %V 202 %X In this work, we propose a novel prior learning method for advancing generalization and uncertainty estimation in deep neural networks. The key idea is to exploit scalable and structured posteriors of neural networks as informative priors with generalization guarantees. Our learned priors provide expressive probabilistic representations at large scale, like Bayesian counterparts of pre-trained models on ImageNet, and further produce non-vacuous generalization bounds. We also extend this idea to a continual learning framework, where the favorable properties of our priors are desirable. Major enablers are our technical contributions: (1) the sums-of-Kronecker-product computations, and (2) the derivations and optimizations of tractable objectives that lead to improved generalization bounds. Empirically, we exhaustively show the effectiveness of this method for uncertainty estimation and generalization.
APA
Schnaus, D., Lee, J., Cremers, D. & Triebel, R.. (2023). Learning Expressive Priors for Generalization and Uncertainty Estimation in Neural Networks. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:30252-30284 Available from https://proceedings.mlr.press/v202/schnaus23a.html.

Related Material