An Analytic Solution to Covariance Propagation in Neural Networks

Oren Wright, Yorie Nakahira, José M. F. Moura
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, PMLR 238:4087-4095, 2024.

Abstract

Uncertainty quantification of neural networks is critical to measuring the reliability and robustness of deep learning systems. However, this often involves costly or inaccurate sampling methods and approximations. This paper presents a sample-free moment propagation technique that propagates mean vectors and covariance matrices across a network to accurately characterize the input-output distributions of neural networks. A key enabler of our technique is an analytic solution for the covariance of random variables passed through nonlinear activation functions, such as Heaviside, ReLU, and GELU. The wide applicability and merits of the proposed technique are shown in experiments analyzing the input-output distributions of trained neural networks and training Bayesian neural networks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v238-wright24a, title = {An Analytic Solution to Covariance Propagation in Neural Networks}, author = {Wright, Oren and Nakahira, Yorie and M. F. Moura, Jos\'{e}}, booktitle = {Proceedings of The 27th International Conference on Artificial Intelligence and Statistics}, pages = {4087--4095}, year = {2024}, editor = {Dasgupta, Sanjoy and Mandt, Stephan and Li, Yingzhen}, volume = {238}, series = {Proceedings of Machine Learning Research}, month = {02--04 May}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v238/wright24a/wright24a.pdf}, url = {https://proceedings.mlr.press/v238/wright24a.html}, abstract = {Uncertainty quantification of neural networks is critical to measuring the reliability and robustness of deep learning systems. However, this often involves costly or inaccurate sampling methods and approximations. This paper presents a sample-free moment propagation technique that propagates mean vectors and covariance matrices across a network to accurately characterize the input-output distributions of neural networks. A key enabler of our technique is an analytic solution for the covariance of random variables passed through nonlinear activation functions, such as Heaviside, ReLU, and GELU. The wide applicability and merits of the proposed technique are shown in experiments analyzing the input-output distributions of trained neural networks and training Bayesian neural networks.} }
Endnote
%0 Conference Paper %T An Analytic Solution to Covariance Propagation in Neural Networks %A Oren Wright %A Yorie Nakahira %A José M. F. Moura %B Proceedings of The 27th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2024 %E Sanjoy Dasgupta %E Stephan Mandt %E Yingzhen Li %F pmlr-v238-wright24a %I PMLR %P 4087--4095 %U https://proceedings.mlr.press/v238/wright24a.html %V 238 %X Uncertainty quantification of neural networks is critical to measuring the reliability and robustness of deep learning systems. However, this often involves costly or inaccurate sampling methods and approximations. This paper presents a sample-free moment propagation technique that propagates mean vectors and covariance matrices across a network to accurately characterize the input-output distributions of neural networks. A key enabler of our technique is an analytic solution for the covariance of random variables passed through nonlinear activation functions, such as Heaviside, ReLU, and GELU. The wide applicability and merits of the proposed technique are shown in experiments analyzing the input-output distributions of trained neural networks and training Bayesian neural networks.
APA
Wright, O., Nakahira, Y. & M. F. Moura, J.. (2024). An Analytic Solution to Covariance Propagation in Neural Networks. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 238:4087-4095 Available from https://proceedings.mlr.press/v238/wright24a.html.

Related Material