Fine-Tuning with Uncertainty-Aware Priors Makes Vision and Language Foundation Models More Reliable

Tim G. J. Rudner, Xiang Pan, Yucen Lily Li, Ravid Shwartz-Ziv, Andrew Gordon Wilson
Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, PMLR 258:154-162, 2025.

Abstract

Fine-tuning off-the-shelf pre-trained neural networks has become the default starting point for a wide range of challenging prediction tasks—especially in computer vision and natural language processing, where pre-trained models trained on millions or even billions of data points are publicly available and can be fine-tuned with a moderate compute budget. However, while fine-tuned models have been shown to significantly improve predictive performance compared to models trained from scratch, they can exhibit poor calibration and fail to reliably identify challenging distribution shifts. In this paper, we improve uncertainty quantification in fine-tuned models by constructing a data-driven uncertainty-aware fine-tuning prior that assigns high probability density to parameters that induce predictive functions with high uncertainty on input points that are meaningfully different from the data. We derive a tractable variational objective to perform approximate inference in models with data-driven uncertainty-aware priors and evaluate models fine-tuned with such priors on different transfer learning tasks. We show that fine-tuning with uncertainty-aware priors significantly improves calibration, selective prediction, and semantic shift detection on computer vision and natural language classification tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v258-rudner25a, title = {Fine-Tuning with Uncertainty-Aware Priors Makes Vision and Language Foundation Models More Reliable}, author = {Rudner, Tim G. J. and Pan, Xiang and Li, Yucen Lily and Shwartz-Ziv, Ravid and Wilson, Andrew Gordon}, booktitle = {Proceedings of The 28th International Conference on Artificial Intelligence and Statistics}, pages = {154--162}, year = {2025}, editor = {Li, Yingzhen and Mandt, Stephan and Agrawal, Shipra and Khan, Emtiyaz}, volume = {258}, series = {Proceedings of Machine Learning Research}, month = {03--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v258/main/assets/rudner25a/rudner25a.pdf}, url = {https://proceedings.mlr.press/v258/rudner25a.html}, abstract = {Fine-tuning off-the-shelf pre-trained neural networks has become the default starting point for a wide range of challenging prediction tasks—especially in computer vision and natural language processing, where pre-trained models trained on millions or even billions of data points are publicly available and can be fine-tuned with a moderate compute budget. However, while fine-tuned models have been shown to significantly improve predictive performance compared to models trained from scratch, they can exhibit poor calibration and fail to reliably identify challenging distribution shifts. In this paper, we improve uncertainty quantification in fine-tuned models by constructing a data-driven uncertainty-aware fine-tuning prior that assigns high probability density to parameters that induce predictive functions with high uncertainty on input points that are meaningfully different from the data. We derive a tractable variational objective to perform approximate inference in models with data-driven uncertainty-aware priors and evaluate models fine-tuned with such priors on different transfer learning tasks. We show that fine-tuning with uncertainty-aware priors significantly improves calibration, selective prediction, and semantic shift detection on computer vision and natural language classification tasks.} }
Endnote
%0 Conference Paper %T Fine-Tuning with Uncertainty-Aware Priors Makes Vision and Language Foundation Models More Reliable %A Tim G. J. Rudner %A Xiang Pan %A Yucen Lily Li %A Ravid Shwartz-Ziv %A Andrew Gordon Wilson %B Proceedings of The 28th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2025 %E Yingzhen Li %E Stephan Mandt %E Shipra Agrawal %E Emtiyaz Khan %F pmlr-v258-rudner25a %I PMLR %P 154--162 %U https://proceedings.mlr.press/v258/rudner25a.html %V 258 %X Fine-tuning off-the-shelf pre-trained neural networks has become the default starting point for a wide range of challenging prediction tasks—especially in computer vision and natural language processing, where pre-trained models trained on millions or even billions of data points are publicly available and can be fine-tuned with a moderate compute budget. However, while fine-tuned models have been shown to significantly improve predictive performance compared to models trained from scratch, they can exhibit poor calibration and fail to reliably identify challenging distribution shifts. In this paper, we improve uncertainty quantification in fine-tuned models by constructing a data-driven uncertainty-aware fine-tuning prior that assigns high probability density to parameters that induce predictive functions with high uncertainty on input points that are meaningfully different from the data. We derive a tractable variational objective to perform approximate inference in models with data-driven uncertainty-aware priors and evaluate models fine-tuned with such priors on different transfer learning tasks. We show that fine-tuning with uncertainty-aware priors significantly improves calibration, selective prediction, and semantic shift detection on computer vision and natural language classification tasks.
APA
Rudner, T.G.J., Pan, X., Li, Y.L., Shwartz-Ziv, R. & Wilson, A.G.. (2025). Fine-Tuning with Uncertainty-Aware Priors Makes Vision and Language Foundation Models More Reliable. Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 258:154-162 Available from https://proceedings.mlr.press/v258/rudner25a.html.

Related Material