HighDimensional VarianceReduced Stochastic Gradient ExpectationMaximization Algorithm
[edit]
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:41804188, 2017.
Abstract
We propose a generic stochastic expectationmaximization (EM) algorithm for the estimation of highdimensional latent variable models. At the core of our algorithm is a novel semistochastic variancereduced gradient designed for the $Q$function in the EM algorithm. Under a mild condition on the initialization, our algorithm is guaranteed to attain a linear convergence rate to the unknown parameter of the latent variable model, and achieve an optimal statistical rate up to a logarithmic factor for parameter estimation. Compared with existing highdimensional EM algorithms, our algorithm enjoys a better computational complexity and is therefore more efficient. We apply our generic algorithm to two illustrative latent variable models: Gaussian mixture model and mixture of linear regression, and demonstrate the advantages of our algorithm by both theoretical analysis and numerical experiments. We believe that the proposed semistochastic gradient is of independent interest for general nonconvex optimization problems with bivariate structures.
Related Material


