Robust Predictive Uncertainty and Double Descent in Contaminated Bayesian Random Features

Michele Caprio, Katerina Papagiannouli, Siu Lun Chau, Sayan Mukherjee
Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, PMLR 337:943-960, 2026.

Abstract

We propose a robust {Bayesian} formulation of random feature (RF) regression that accounts explicitly for prior and likelihood misspecification via {Huber}-style contamination sets. Starting from the classical equivalence between ridge-regularized RF training and {Bayesian} inference with {Gaussian} priors and likelihoods, we replace the single prior and likelihood with $\epsilon$- and $\eta$-contaminated credal sets, respectively, and perform inference using pessimistic generalized {Bayesian} updating. We derive explicit and tractable bounds for the resulting lower and upper posterior predictive densities. These bounds show that, when contamination is moderate, prior and likelihood ambiguity effectively acts as a direct contamination of the posterior predictive distribution, yielding uncertainty envelopes around the classical {Gaussian} predictive. We introduce an Imprecise Highest Density Region ({IHDR}) for robust predictive uncertainty quantification and show that it admits an efficient outer approximation via an adjusted {Gaussian} credible interval. We further obtain predictive variance bounds (under a mild truncation approximation for the upper bound) and prove that they preserve the leading-order proportional-growth asymptotics known for RF models. Together, these results establish a robustness theory for {Bayesian} random features: predictive uncertainty remains computationally tractable, inherits the classical double-descent phase structure, and is improved by explicit worst-case guarantees under bounded prior and likelihood misspecification.

Cite this Paper

BibTeX
@InProceedings{pmlr-v337-caprio26a, title = {Robust Predictive Uncertainty and Double Descent in Contaminated {Bayesian} Random Features}, author = {Caprio, Michele and Papagiannouli, Katerina and Chau, Siu Lun and Mukherjee, Sayan}, booktitle = {Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence}, pages = {943--960}, year = {2026}, editor = {Perković, Emilija and Malinsky, Daniel}, volume = {337}, series = {Proceedings of Machine Learning Research}, month = {17--21 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v337/main/assets/caprio26a/caprio26a.pdf}, url = {https://proceedings.mlr.press/v337/caprio26a.html}, abstract = {We propose a robust {Bayesian} formulation of random feature (RF) regression that accounts explicitly for prior and likelihood misspecification via {Huber}-style contamination sets. Starting from the classical equivalence between ridge-regularized RF training and {Bayesian} inference with {Gaussian} priors and likelihoods, we replace the single prior and likelihood with $\epsilon$- and $\eta$-contaminated credal sets, respectively, and perform inference using pessimistic generalized {Bayesian} updating. We derive explicit and tractable bounds for the resulting lower and upper posterior predictive densities. These bounds show that, when contamination is moderate, prior and likelihood ambiguity effectively acts as a direct contamination of the posterior predictive distribution, yielding uncertainty envelopes around the classical {Gaussian} predictive. We introduce an Imprecise Highest Density Region ({IHDR}) for robust predictive uncertainty quantification and show that it admits an efficient outer approximation via an adjusted {Gaussian} credible interval. We further obtain predictive variance bounds (under a mild truncation approximation for the upper bound) and prove that they preserve the leading-order proportional-growth asymptotics known for RF models. Together, these results establish a robustness theory for {Bayesian} random features: predictive uncertainty remains computationally tractable, inherits the classical double-descent phase structure, and is improved by explicit worst-case guarantees under bounded prior and likelihood misspecification.} }
Endnote
%0 Conference Paper %T Robust Predictive Uncertainty and Double Descent in Contaminated Bayesian Random Features %A Michele Caprio %A Katerina Papagiannouli %A Siu Lun Chau %A Sayan Mukherjee %B Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2026 %E Emilija Perković %E Daniel Malinsky %F pmlr-v337-caprio26a %I PMLR %P 943--960 %U https://proceedings.mlr.press/v337/caprio26a.html %V 337 %X We propose a robust {Bayesian} formulation of random feature (RF) regression that accounts explicitly for prior and likelihood misspecification via {Huber}-style contamination sets. Starting from the classical equivalence between ridge-regularized RF training and {Bayesian} inference with {Gaussian} priors and likelihoods, we replace the single prior and likelihood with $\epsilon$- and $\eta$-contaminated credal sets, respectively, and perform inference using pessimistic generalized {Bayesian} updating. We derive explicit and tractable bounds for the resulting lower and upper posterior predictive densities. These bounds show that, when contamination is moderate, prior and likelihood ambiguity effectively acts as a direct contamination of the posterior predictive distribution, yielding uncertainty envelopes around the classical {Gaussian} predictive. We introduce an Imprecise Highest Density Region ({IHDR}) for robust predictive uncertainty quantification and show that it admits an efficient outer approximation via an adjusted {Gaussian} credible interval. We further obtain predictive variance bounds (under a mild truncation approximation for the upper bound) and prove that they preserve the leading-order proportional-growth asymptotics known for RF models. Together, these results establish a robustness theory for {Bayesian} random features: predictive uncertainty remains computationally tractable, inherits the classical double-descent phase structure, and is improved by explicit worst-case guarantees under bounded prior and likelihood misspecification.
APA
Caprio, M., Papagiannouli, K., Chau, S.L. & Mukherjee, S.. (2026). Robust Predictive Uncertainty and Double Descent in Contaminated Bayesian Random Features. Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 337:943-960 Available from https://proceedings.mlr.press/v337/caprio26a.html.

Related Material