Efficient Approximate Inference for Stationary Kernel on Frequency Domain

Yohan Jung, Kyungwoo Song, Jinkyoo Park
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:10502-10538, 2022.

Abstract

Based on the Fourier duality between a stationary kernel and its spectral density, modeling the spectral density using a Gaussian mixture density enables one to construct a flexible kernel, known as a Spectral Mixture kernel, that can model any stationary kernel. However, despite its expressive power, training this kernel is typically difficult because scalability and overfitting issues often arise due to a large number of training parameters. To resolve these issues, we propose an approximate inference method for estimating the Spectral mixture kernel hyperparameters. Specifically, we approximate this kernel by using the finite random spectral points based on Random Fourier Feature and optimize the parameters for the distribution of spectral points by sampling-based variational inference. To improve this inference procedure, we analyze the training loss and propose two special methods: a sampling method of spectral points to reduce the error of the approximate kernel in training, and an approximate natural gradient to accelerate the convergence of parameter inference.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-jung22b, title = {Efficient Approximate Inference for Stationary Kernel on Frequency Domain}, author = {Jung, Yohan and Song, Kyungwoo and Park, Jinkyoo}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {10502--10538}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/jung22b/jung22b.pdf}, url = {https://proceedings.mlr.press/v162/jung22b.html}, abstract = {Based on the Fourier duality between a stationary kernel and its spectral density, modeling the spectral density using a Gaussian mixture density enables one to construct a flexible kernel, known as a Spectral Mixture kernel, that can model any stationary kernel. However, despite its expressive power, training this kernel is typically difficult because scalability and overfitting issues often arise due to a large number of training parameters. To resolve these issues, we propose an approximate inference method for estimating the Spectral mixture kernel hyperparameters. Specifically, we approximate this kernel by using the finite random spectral points based on Random Fourier Feature and optimize the parameters for the distribution of spectral points by sampling-based variational inference. To improve this inference procedure, we analyze the training loss and propose two special methods: a sampling method of spectral points to reduce the error of the approximate kernel in training, and an approximate natural gradient to accelerate the convergence of parameter inference.} }
Endnote
%0 Conference Paper %T Efficient Approximate Inference for Stationary Kernel on Frequency Domain %A Yohan Jung %A Kyungwoo Song %A Jinkyoo Park %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-jung22b %I PMLR %P 10502--10538 %U https://proceedings.mlr.press/v162/jung22b.html %V 162 %X Based on the Fourier duality between a stationary kernel and its spectral density, modeling the spectral density using a Gaussian mixture density enables one to construct a flexible kernel, known as a Spectral Mixture kernel, that can model any stationary kernel. However, despite its expressive power, training this kernel is typically difficult because scalability and overfitting issues often arise due to a large number of training parameters. To resolve these issues, we propose an approximate inference method for estimating the Spectral mixture kernel hyperparameters. Specifically, we approximate this kernel by using the finite random spectral points based on Random Fourier Feature and optimize the parameters for the distribution of spectral points by sampling-based variational inference. To improve this inference procedure, we analyze the training loss and propose two special methods: a sampling method of spectral points to reduce the error of the approximate kernel in training, and an approximate natural gradient to accelerate the convergence of parameter inference.
APA
Jung, Y., Song, K. & Park, J.. (2022). Efficient Approximate Inference for Stationary Kernel on Frequency Domain. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:10502-10538 Available from https://proceedings.mlr.press/v162/jung22b.html.

Related Material