On the Equivalence of Random Network Distillation, Deep Ensembles, and Bayesian Inference

Moritz Akiya Zanger, Yijun Wu, Pascal R. van der Vaart, Wendelin Böhmer, Matthijs T. J. Spaan
Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, PMLR 337:7952-7976, 2026.

Abstract

Uncertainty quantification is central to safe and efficient deployments of deep learning models, yet many computationally practical methods lack lacking rigorous theoretical motivation. Random network distillation (RND) is a lightweight technique that measures novelty via prediction errors against a fixed random target. While empirically effective, it has remained unclear what uncertainties RND measures and how its estimates relate to other approaches, e.g., {Bayesian} inference or deep ensembles. We establish these missing theoretical connections by analyzing RND within the neural tangent kernel framework in the limit of infinite network width. Our analysis reveals two central findings in this limit: (1) The uncertainty signal from RND—its squared self-predictive error—is equivalent to the predictive variance of a deep ensemble. (2) By constructing a specific RND target function, we show that the RND error distribution can be made to mirror the centered posterior predictive distribution of {Bayesian} inference with wide neural networks. Based on this equivalence, we moreover devise a posterior sampling algorithm that generates i.i.d. samples from an exact {Bayesian} posterior predictive distribution using this modified \textit{{Bayesian} RND} model. Collectively, our findings provide a unified theoretical perspective that places RND within the principled frameworks of deep ensembles and {Bayesian} inference, and offer new avenues for efficient yet theoretically grounded uncertainty quantification methods.

Cite this Paper

BibTeX
@InProceedings{pmlr-v337-zanger26a, title = {On the Equivalence of Random Network Distillation, Deep Ensembles, and {Bayesian} Inference}, author = {Zanger, Moritz Akiya and Wu, Yijun and van der Vaart, Pascal R. and B\"{o}hmer, Wendelin and Spaan, Matthijs T. J.}, booktitle = {Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence}, pages = {7952--7976}, year = {2026}, editor = {Perković, Emilija and Malinsky, Daniel}, volume = {337}, series = {Proceedings of Machine Learning Research}, month = {17--21 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v337/main/assets/zanger26a/zanger26a.pdf}, url = {https://proceedings.mlr.press/v337/zanger26a.html}, abstract = {Uncertainty quantification is central to safe and efficient deployments of deep learning models, yet many computationally practical methods lack lacking rigorous theoretical motivation. Random network distillation (RND) is a lightweight technique that measures novelty via prediction errors against a fixed random target. While empirically effective, it has remained unclear what uncertainties RND measures and how its estimates relate to other approaches, e.g., {Bayesian} inference or deep ensembles. We establish these missing theoretical connections by analyzing RND within the neural tangent kernel framework in the limit of infinite network width. Our analysis reveals two central findings in this limit: (1) The uncertainty signal from RND—its squared self-predictive error—is equivalent to the predictive variance of a deep ensemble. (2) By constructing a specific RND target function, we show that the RND error distribution can be made to mirror the centered posterior predictive distribution of {Bayesian} inference with wide neural networks. Based on this equivalence, we moreover devise a posterior sampling algorithm that generates i.i.d. samples from an exact {Bayesian} posterior predictive distribution using this modified \textit{{Bayesian} RND} model. Collectively, our findings provide a unified theoretical perspective that places RND within the principled frameworks of deep ensembles and {Bayesian} inference, and offer new avenues for efficient yet theoretically grounded uncertainty quantification methods.} }
Endnote
%0 Conference Paper %T On the Equivalence of Random Network Distillation, Deep Ensembles, and Bayesian Inference %A Moritz Akiya Zanger %A Yijun Wu %A Pascal R. van der Vaart %A Wendelin Böhmer %A Matthijs T. J. Spaan %B Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2026 %E Emilija Perković %E Daniel Malinsky %F pmlr-v337-zanger26a %I PMLR %P 7952--7976 %U https://proceedings.mlr.press/v337/zanger26a.html %V 337 %X Uncertainty quantification is central to safe and efficient deployments of deep learning models, yet many computationally practical methods lack lacking rigorous theoretical motivation. Random network distillation (RND) is a lightweight technique that measures novelty via prediction errors against a fixed random target. While empirically effective, it has remained unclear what uncertainties RND measures and how its estimates relate to other approaches, e.g., {Bayesian} inference or deep ensembles. We establish these missing theoretical connections by analyzing RND within the neural tangent kernel framework in the limit of infinite network width. Our analysis reveals two central findings in this limit: (1) The uncertainty signal from RND—its squared self-predictive error—is equivalent to the predictive variance of a deep ensemble. (2) By constructing a specific RND target function, we show that the RND error distribution can be made to mirror the centered posterior predictive distribution of {Bayesian} inference with wide neural networks. Based on this equivalence, we moreover devise a posterior sampling algorithm that generates i.i.d. samples from an exact {Bayesian} posterior predictive distribution using this modified \textit{{Bayesian} RND} model. Collectively, our findings provide a unified theoretical perspective that places RND within the principled frameworks of deep ensembles and {Bayesian} inference, and offer new avenues for efficient yet theoretically grounded uncertainty quantification methods.
APA
Zanger, M.A., Wu, Y., van der Vaart, P.R., Böhmer, W. & Spaan, M.T.J.. (2026). On the Equivalence of Random Network Distillation, Deep Ensembles, and Bayesian Inference. Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 337:7952-7976 Available from https://proceedings.mlr.press/v337/zanger26a.html.

Related Material