Bias-Free Scalable Gaussian Processes via Randomized Truncations

Andres Potapczynski, Luhuan Wu, Dan Biderman, Geoff Pleiss, John P Cunningham
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:8609-8619, 2021.

Abstract

Scalable Gaussian Process methods are computationally attractive, yet introduce modeling biases that require rigorous study. This paper analyzes two common techniques: early truncated conjugate gradients (CG) and random Fourier features (RFF). We find that both methods introduce a systematic bias on the learned hyperparameters: CG tends to underfit while RFF tends to overfit. We address these issues using randomized truncation estimators that eliminate bias in exchange for increased variance. In the case of RFF, we show that the bias-to-variance conversion is indeed a trade-off: the additional variance proves detrimental to optimization. However, in the case of CG, our unbiased learning procedure meaningfully outperforms its biased counterpart with minimal additional computation. Our code is available at https://github.com/ cunningham-lab/RTGPS.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-potapczynski21a, title = {Bias-Free Scalable Gaussian Processes via Randomized Truncations}, author = {Potapczynski, Andres and Wu, Luhuan and Biderman, Dan and Pleiss, Geoff and Cunningham, John P}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {8609--8619}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/potapczynski21a/potapczynski21a.pdf}, url = {https://proceedings.mlr.press/v139/potapczynski21a.html}, abstract = {Scalable Gaussian Process methods are computationally attractive, yet introduce modeling biases that require rigorous study. This paper analyzes two common techniques: early truncated conjugate gradients (CG) and random Fourier features (RFF). We find that both methods introduce a systematic bias on the learned hyperparameters: CG tends to underfit while RFF tends to overfit. We address these issues using randomized truncation estimators that eliminate bias in exchange for increased variance. In the case of RFF, we show that the bias-to-variance conversion is indeed a trade-off: the additional variance proves detrimental to optimization. However, in the case of CG, our unbiased learning procedure meaningfully outperforms its biased counterpart with minimal additional computation. Our code is available at https://github.com/ cunningham-lab/RTGPS.} }
Endnote
%0 Conference Paper %T Bias-Free Scalable Gaussian Processes via Randomized Truncations %A Andres Potapczynski %A Luhuan Wu %A Dan Biderman %A Geoff Pleiss %A John P Cunningham %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-potapczynski21a %I PMLR %P 8609--8619 %U https://proceedings.mlr.press/v139/potapczynski21a.html %V 139 %X Scalable Gaussian Process methods are computationally attractive, yet introduce modeling biases that require rigorous study. This paper analyzes two common techniques: early truncated conjugate gradients (CG) and random Fourier features (RFF). We find that both methods introduce a systematic bias on the learned hyperparameters: CG tends to underfit while RFF tends to overfit. We address these issues using randomized truncation estimators that eliminate bias in exchange for increased variance. In the case of RFF, we show that the bias-to-variance conversion is indeed a trade-off: the additional variance proves detrimental to optimization. However, in the case of CG, our unbiased learning procedure meaningfully outperforms its biased counterpart with minimal additional computation. Our code is available at https://github.com/ cunningham-lab/RTGPS.
APA
Potapczynski, A., Wu, L., Biderman, D., Pleiss, G. & Cunningham, J.P.. (2021). Bias-Free Scalable Gaussian Processes via Randomized Truncations. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:8609-8619 Available from https://proceedings.mlr.press/v139/potapczynski21a.html.

Related Material