Adaptive Riemannian stochastic gradient descent and reparameterization for Gaussian mixture model fitting

Chunlin Ji, Yuhao Fu, Ping He
Proceedings of the 15th Asian Conference on Machine Learning, PMLR 222:534-549, 2024.

Abstract

Recent advances in manifold optimization for the Gaussian mixture model (GMM) have gained increasing interest. In this work, instead of directly addressing the manifold optimization on covariance matrices of GMM, we consider the GMM fitting as an optimization of the density function over a statistical manifold and seek the natural gradient to speed up the optimization process. We present an upper bound for the Kullback–Leibler (KL) divergence between two GMMs and obtain simple closed-form expressions for the natural gradients. With the natural gradients, we then apply the Riemannian stochastic gradient descent (RSGD) algorithm to optimize covariance matrices on a symmetric and positive definite (SPD) matrix manifold. We further propose a Riemannian Adam (RAdam) algorithm that extends the momentum method and adaptive learning in the Euclidean space to the SPD manifold space. Extensive simulations show that the proposed algorithms scale well to high-dimensional large-scale datasets and outperform expectation maximization (EM) algorithms in fitted log-likelihood.

Cite this Paper

BibTeX
@InProceedings{pmlr-v222-ji24a, title = {Adaptive {R}iemannian stochastic gradient descent and reparameterization for {G}aussian mixture model fitting}, author = {Ji, Chunlin and Fu, Yuhao and He, Ping}, booktitle = {Proceedings of the 15th Asian Conference on Machine Learning}, pages = {534--549}, year = {2024}, editor = {Yanıkoğlu, Berrin and Buntine, Wray}, volume = {222}, series = {Proceedings of Machine Learning Research}, month = {11--14 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v222/ji24a/ji24a.pdf}, url = {https://proceedings.mlr.press/v222/ji24a.html}, abstract = {Recent advances in manifold optimization for the Gaussian mixture model (GMM) have gained increasing interest. In this work, instead of directly addressing the manifold optimization on covariance matrices of GMM, we consider the GMM fitting as an optimization of the density function over a statistical manifold and seek the natural gradient to speed up the optimization process. We present an upper bound for the Kullback–Leibler (KL) divergence between two GMMs and obtain simple closed-form expressions for the natural gradients. With the natural gradients, we then apply the Riemannian stochastic gradient descent (RSGD) algorithm to optimize covariance matrices on a symmetric and positive definite (SPD) matrix manifold. We further propose a Riemannian Adam (RAdam) algorithm that extends the momentum method and adaptive learning in the Euclidean space to the SPD manifold space. Extensive simulations show that the proposed algorithms scale well to high-dimensional large-scale datasets and outperform expectation maximization (EM) algorithms in fitted log-likelihood.} }
Endnote
%0 Conference Paper %T Adaptive Riemannian stochastic gradient descent and reparameterization for Gaussian mixture model fitting %A Chunlin Ji %A Yuhao Fu %A Ping He %B Proceedings of the 15th Asian Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Berrin Yanıkoğlu %E Wray Buntine %F pmlr-v222-ji24a %I PMLR %P 534--549 %U https://proceedings.mlr.press/v222/ji24a.html %V 222 %X Recent advances in manifold optimization for the Gaussian mixture model (GMM) have gained increasing interest. In this work, instead of directly addressing the manifold optimization on covariance matrices of GMM, we consider the GMM fitting as an optimization of the density function over a statistical manifold and seek the natural gradient to speed up the optimization process. We present an upper bound for the Kullback–Leibler (KL) divergence between two GMMs and obtain simple closed-form expressions for the natural gradients. With the natural gradients, we then apply the Riemannian stochastic gradient descent (RSGD) algorithm to optimize covariance matrices on a symmetric and positive definite (SPD) matrix manifold. We further propose a Riemannian Adam (RAdam) algorithm that extends the momentum method and adaptive learning in the Euclidean space to the SPD manifold space. Extensive simulations show that the proposed algorithms scale well to high-dimensional large-scale datasets and outperform expectation maximization (EM) algorithms in fitted log-likelihood.
APA
Ji, C., Fu, Y. & He, P.. (2024). Adaptive Riemannian stochastic gradient descent and reparameterization for Gaussian mixture model fitting. Proceedings of the 15th Asian Conference on Machine Learning, in Proceedings of Machine Learning Research 222:534-549 Available from https://proceedings.mlr.press/v222/ji24a.html.

Related Material