Online Partial Least Square Optimization: Dropping Convexity for Better Efficiency and Scalability

Zhehui Chen, Lin F. Yang, Chris Junchi Li, Tuo Zhao
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:777-786, 2017.

Abstract

Multiview representation learning is popular for latent factor analysis. Many existing approaches formulate the multiview representation learning as convex optimization problems, where global optima can be obtained by certain algorithms in polynomial time. However, many evidences have corroborated that heuristic nonconvex approaches also have good empirical computational performance and convergence to the global optima, although there is a lack of theoretical justification. Such a gap between theory and practice motivates us to study a nonconvex formulation for multiview representation learning, which can be efficiently solved by a simple stochastic gradient descent method. By analyzing the dynamics of the algorithm based on diffusion processes, we establish a global rate of convergence to the global optima. Numerical experiments are provided to support our theory.

Cite this Paper

BibTeX
@InProceedings{pmlr-v70-chen17h, title = {Online Partial Least Square Optimization: Dropping Convexity for Better Efficiency and Scalability}, author = {Zhehui Chen and Lin F. Yang and Chris Junchi Li and Tuo Zhao}, booktitle = {Proceedings of the 34th International Conference on Machine Learning}, pages = {777--786}, year = {2017}, editor = {Precup, Doina and Teh, Yee Whye}, volume = {70}, series = {Proceedings of Machine Learning Research}, month = {06--11 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v70/chen17h/chen17h.pdf}, url = {https://proceedings.mlr.press/v70/chen17h.html}, abstract = {Multiview representation learning is popular for latent factor analysis. Many existing approaches formulate the multiview representation learning as convex optimization problems, where global optima can be obtained by certain algorithms in polynomial time. However, many evidences have corroborated that heuristic nonconvex approaches also have good empirical computational performance and convergence to the global optima, although there is a lack of theoretical justification. Such a gap between theory and practice motivates us to study a nonconvex formulation for multiview representation learning, which can be efficiently solved by a simple stochastic gradient descent method. By analyzing the dynamics of the algorithm based on diffusion processes, we establish a global rate of convergence to the global optima. Numerical experiments are provided to support our theory.} }
Endnote
%0 Conference Paper %T Online Partial Least Square Optimization: Dropping Convexity for Better Efficiency and Scalability %A Zhehui Chen %A Lin F. Yang %A Chris Junchi Li %A Tuo Zhao %B Proceedings of the 34th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Doina Precup %E Yee Whye Teh %F pmlr-v70-chen17h %I PMLR %P 777--786 %U https://proceedings.mlr.press/v70/chen17h.html %V 70 %X Multiview representation learning is popular for latent factor analysis. Many existing approaches formulate the multiview representation learning as convex optimization problems, where global optima can be obtained by certain algorithms in polynomial time. However, many evidences have corroborated that heuristic nonconvex approaches also have good empirical computational performance and convergence to the global optima, although there is a lack of theoretical justification. Such a gap between theory and practice motivates us to study a nonconvex formulation for multiview representation learning, which can be efficiently solved by a simple stochastic gradient descent method. By analyzing the dynamics of the algorithm based on diffusion processes, we establish a global rate of convergence to the global optima. Numerical experiments are provided to support our theory.
APA
Chen, Z., Yang, L.F., Li, C.J. & Zhao, T.. (2017). Online Partial Least Square Optimization: Dropping Convexity for Better Efficiency and Scalability. Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research 70:777-786 Available from https://proceedings.mlr.press/v70/chen17h.html.

Related Material