The Fine Print on Tempered Posteriors

Konstantinos Pitas, Julyan Arbel
Proceedings of the 15th Asian Conference on Machine Learning, PMLR 222:1087-1102, 2024.

Abstract

We conduct a detailed investigation of tempered posteriors and uncover a number of crucial and previously undiscussed points. Contrary to previous results, we first show that for realistic models and datasets and the tightly controlled case of the Laplace approximation to the posterior, stochasticity does not in general improve test accuracy. The coldest temperature is often optimal. One might think that Bayesian models with some stochasticity can at least obtain improvements in terms of calibration. However, we show empirically that when gains are obtained this comes at the cost of degradation in test accuracy. We then discuss how targeting Frequentist metrics using Bayesian models provides a simple explanation of the need for a temperature parameter $\lambda$ in the optimization objective. Contrary to prior works, we finally show through a PAC-Bayesian analysis that the temperature $\lambda$ cannot be seen as simply fixing a misspecified prior or likelihood.

Cite this Paper

BibTeX
@InProceedings{pmlr-v222-pitas24a, title = {The Fine Print on Tempered Posteriors}, author = {Pitas, Konstantinos and Arbel, Julyan}, booktitle = {Proceedings of the 15th Asian Conference on Machine Learning}, pages = {1087--1102}, year = {2024}, editor = {Yanıkoğlu, Berrin and Buntine, Wray}, volume = {222}, series = {Proceedings of Machine Learning Research}, month = {11--14 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v222/pitas24a/pitas24a.pdf}, url = {https://proceedings.mlr.press/v222/pitas24a.html}, abstract = {We conduct a detailed investigation of tempered posteriors and uncover a number of crucial and previously undiscussed points. Contrary to previous results, we first show that for realistic models and datasets and the tightly controlled case of the Laplace approximation to the posterior, stochasticity does not in general improve test accuracy. The coldest temperature is often optimal. One might think that Bayesian models with some stochasticity can at least obtain improvements in terms of calibration. However, we show empirically that when gains are obtained this comes at the cost of degradation in test accuracy. We then discuss how targeting Frequentist metrics using Bayesian models provides a simple explanation of the need for a temperature parameter $\lambda$ in the optimization objective. Contrary to prior works, we finally show through a PAC-Bayesian analysis that the temperature $\lambda$ cannot be seen as simply fixing a misspecified prior or likelihood.} }
Endnote
%0 Conference Paper %T The Fine Print on Tempered Posteriors %A Konstantinos Pitas %A Julyan Arbel %B Proceedings of the 15th Asian Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Berrin Yanıkoğlu %E Wray Buntine %F pmlr-v222-pitas24a %I PMLR %P 1087--1102 %U https://proceedings.mlr.press/v222/pitas24a.html %V 222 %X We conduct a detailed investigation of tempered posteriors and uncover a number of crucial and previously undiscussed points. Contrary to previous results, we first show that for realistic models and datasets and the tightly controlled case of the Laplace approximation to the posterior, stochasticity does not in general improve test accuracy. The coldest temperature is often optimal. One might think that Bayesian models with some stochasticity can at least obtain improvements in terms of calibration. However, we show empirically that when gains are obtained this comes at the cost of degradation in test accuracy. We then discuss how targeting Frequentist metrics using Bayesian models provides a simple explanation of the need for a temperature parameter $\lambda$ in the optimization objective. Contrary to prior works, we finally show through a PAC-Bayesian analysis that the temperature $\lambda$ cannot be seen as simply fixing a misspecified prior or likelihood.
APA
Pitas, K. & Arbel, J.. (2024). The Fine Print on Tempered Posteriors. Proceedings of the 15th Asian Conference on Machine Learning, in Proceedings of Machine Learning Research 222:1087-1102 Available from https://proceedings.mlr.press/v222/pitas24a.html.

Related Material