Efficient preconditioned stochastic gradient descent for estimation in latent variable models

Charlotte Baey, Maud Delattre, Estelle Kuhn, Jean-Benoist Leger, Sarah Lemler
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:1430-1453, 2023.

Abstract

Latent variable models are powerful tools for modeling complex phenomena involving in particular partially observed data, unobserved variables or underlying complex unknown structures. Inference is often difficult due to the latent structure of the model. To deal with parameter estimation in the presence of latent variables, well-known efficient methods exist, such as gradient-based and EM-type algorithms, but with practical and theoretical limitations. In this paper, we propose as an alternative for parameter estimation an efficient preconditioned stochastic gradient algorithm. Our method includes a preconditioning step based on a positive definite Fisher information matrix estimate. We prove convergence results for the proposed algorithm under mild assumptions for very general latent variables models. We illustrate through relevant simulations the performance of the proposed methodology in a nonlinear mixed effects model and in a stochastic block model.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-baey23a, title = {Efficient preconditioned stochastic gradient descent for estimation in latent variable models}, author = {Baey, Charlotte and Delattre, Maud and Kuhn, Estelle and Leger, Jean-Benoist and Lemler, Sarah}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {1430--1453}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/baey23a/baey23a.pdf}, url = {https://proceedings.mlr.press/v202/baey23a.html}, abstract = {Latent variable models are powerful tools for modeling complex phenomena involving in particular partially observed data, unobserved variables or underlying complex unknown structures. Inference is often difficult due to the latent structure of the model. To deal with parameter estimation in the presence of latent variables, well-known efficient methods exist, such as gradient-based and EM-type algorithms, but with practical and theoretical limitations. In this paper, we propose as an alternative for parameter estimation an efficient preconditioned stochastic gradient algorithm. Our method includes a preconditioning step based on a positive definite Fisher information matrix estimate. We prove convergence results for the proposed algorithm under mild assumptions for very general latent variables models. We illustrate through relevant simulations the performance of the proposed methodology in a nonlinear mixed effects model and in a stochastic block model.} }
Endnote
%0 Conference Paper %T Efficient preconditioned stochastic gradient descent for estimation in latent variable models %A Charlotte Baey %A Maud Delattre %A Estelle Kuhn %A Jean-Benoist Leger %A Sarah Lemler %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-baey23a %I PMLR %P 1430--1453 %U https://proceedings.mlr.press/v202/baey23a.html %V 202 %X Latent variable models are powerful tools for modeling complex phenomena involving in particular partially observed data, unobserved variables or underlying complex unknown structures. Inference is often difficult due to the latent structure of the model. To deal with parameter estimation in the presence of latent variables, well-known efficient methods exist, such as gradient-based and EM-type algorithms, but with practical and theoretical limitations. In this paper, we propose as an alternative for parameter estimation an efficient preconditioned stochastic gradient algorithm. Our method includes a preconditioning step based on a positive definite Fisher information matrix estimate. We prove convergence results for the proposed algorithm under mild assumptions for very general latent variables models. We illustrate through relevant simulations the performance of the proposed methodology in a nonlinear mixed effects model and in a stochastic block model.
APA
Baey, C., Delattre, M., Kuhn, E., Leger, J. & Lemler, S.. (2023). Efficient preconditioned stochastic gradient descent for estimation in latent variable models. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:1430-1453 Available from https://proceedings.mlr.press/v202/baey23a.html.

Related Material