Convergence of Online Learning Algorithm for a Mixture of Multiple Linear Regressions

Yujing Liu, Zhixin Liu, Lei Guo
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:31516-31540, 2024.

Abstract

This paper considers the parameter learning and data clustering problem for MLR with multiple sub-models and arbitrary mixing weights. To deal with the data streaming case, we propose an online learning algorithm to estimate the unknown parameters. By utilizing Ljung’s ODE method, we establish the almost sure convergence results of this MLR problem without the traditional i.i.d. assumption on the input data for the first time. Based on the convergence property and using the classical stochastic Lyapunov function method, we also obtain the convergence rate analysis of the proposed algorithm for the first time. In addition, the data clustering can asymptotically achieve the same performance as the case with known parameters. Future work will consider how to relax the asymptotically stationary and ergodic assumption on the input data, and how to design algorithms with global convergence performance for the MLR problem.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-liu24an, title = {Convergence of Online Learning Algorithm for a Mixture of Multiple Linear Regressions}, author = {Liu, Yujing and Liu, Zhixin and Guo, Lei}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {31516--31540}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/liu24an/liu24an.pdf}, url = {https://proceedings.mlr.press/v235/liu24an.html}, abstract = {This paper considers the parameter learning and data clustering problem for MLR with multiple sub-models and arbitrary mixing weights. To deal with the data streaming case, we propose an online learning algorithm to estimate the unknown parameters. By utilizing Ljung’s ODE method, we establish the almost sure convergence results of this MLR problem without the traditional i.i.d. assumption on the input data for the first time. Based on the convergence property and using the classical stochastic Lyapunov function method, we also obtain the convergence rate analysis of the proposed algorithm for the first time. In addition, the data clustering can asymptotically achieve the same performance as the case with known parameters. Future work will consider how to relax the asymptotically stationary and ergodic assumption on the input data, and how to design algorithms with global convergence performance for the MLR problem.} }
Endnote
%0 Conference Paper %T Convergence of Online Learning Algorithm for a Mixture of Multiple Linear Regressions %A Yujing Liu %A Zhixin Liu %A Lei Guo %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-liu24an %I PMLR %P 31516--31540 %U https://proceedings.mlr.press/v235/liu24an.html %V 235 %X This paper considers the parameter learning and data clustering problem for MLR with multiple sub-models and arbitrary mixing weights. To deal with the data streaming case, we propose an online learning algorithm to estimate the unknown parameters. By utilizing Ljung’s ODE method, we establish the almost sure convergence results of this MLR problem without the traditional i.i.d. assumption on the input data for the first time. Based on the convergence property and using the classical stochastic Lyapunov function method, we also obtain the convergence rate analysis of the proposed algorithm for the first time. In addition, the data clustering can asymptotically achieve the same performance as the case with known parameters. Future work will consider how to relax the asymptotically stationary and ergodic assumption on the input data, and how to design algorithms with global convergence performance for the MLR problem.
APA
Liu, Y., Liu, Z. & Guo, L.. (2024). Convergence of Online Learning Algorithm for a Mixture of Multiple Linear Regressions. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:31516-31540 Available from https://proceedings.mlr.press/v235/liu24an.html.

Related Material