On Signal-to-Noise Ratio Issues in Variational Inference for Deep Gaussian Processes

Tim G. J. Rudner, Oscar Key, Yarin Gal, Tom Rainforth
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:9148-9156, 2021.

Abstract

We show that the gradient estimates used in training Deep Gaussian Processes (DGPs) with importance-weighted variational inference are susceptible to signal-to-noise ratio (SNR) issues. Specifically, we show both theoretically and via an extensive empirical evaluation that the SNR of the gradient estimates for the latent variable’s variational parameters decreases as the number of importance samples increases. As a result, these gradient estimates degrade to pure noise if the number of importance samples is too large. To address this pathology, we show how doubly-reparameterized gradient estimators, originally proposed for training variational autoencoders, can be adapted to the DGP setting and that the resultant estimators completely remedy the SNR issue, thereby providing more reliable training. Finally, we demonstrate that our fix can lead to consistent improvements in the predictive performance of DGP models.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-rudner21a, title = {On Signal-to-Noise Ratio Issues in Variational Inference for Deep Gaussian Processes}, author = {Rudner, Tim G. J. and Key, Oscar and Gal, Yarin and Rainforth, Tom}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {9148--9156}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/rudner21a/rudner21a.pdf}, url = {https://proceedings.mlr.press/v139/rudner21a.html}, abstract = {We show that the gradient estimates used in training Deep Gaussian Processes (DGPs) with importance-weighted variational inference are susceptible to signal-to-noise ratio (SNR) issues. Specifically, we show both theoretically and via an extensive empirical evaluation that the SNR of the gradient estimates for the latent variable’s variational parameters decreases as the number of importance samples increases. As a result, these gradient estimates degrade to pure noise if the number of importance samples is too large. To address this pathology, we show how doubly-reparameterized gradient estimators, originally proposed for training variational autoencoders, can be adapted to the DGP setting and that the resultant estimators completely remedy the SNR issue, thereby providing more reliable training. Finally, we demonstrate that our fix can lead to consistent improvements in the predictive performance of DGP models.} }
Endnote
%0 Conference Paper %T On Signal-to-Noise Ratio Issues in Variational Inference for Deep Gaussian Processes %A Tim G. J. Rudner %A Oscar Key %A Yarin Gal %A Tom Rainforth %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-rudner21a %I PMLR %P 9148--9156 %U https://proceedings.mlr.press/v139/rudner21a.html %V 139 %X We show that the gradient estimates used in training Deep Gaussian Processes (DGPs) with importance-weighted variational inference are susceptible to signal-to-noise ratio (SNR) issues. Specifically, we show both theoretically and via an extensive empirical evaluation that the SNR of the gradient estimates for the latent variable’s variational parameters decreases as the number of importance samples increases. As a result, these gradient estimates degrade to pure noise if the number of importance samples is too large. To address this pathology, we show how doubly-reparameterized gradient estimators, originally proposed for training variational autoencoders, can be adapted to the DGP setting and that the resultant estimators completely remedy the SNR issue, thereby providing more reliable training. Finally, we demonstrate that our fix can lead to consistent improvements in the predictive performance of DGP models.
APA
Rudner, T.G.J., Key, O., Gal, Y. & Rainforth, T.. (2021). On Signal-to-Noise Ratio Issues in Variational Inference for Deep Gaussian Processes. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:9148-9156 Available from https://proceedings.mlr.press/v139/rudner21a.html.

Related Material