Provable Model-Parallel Distributed Principal Component Analysis with Parallel Deflation

Fangshuo Liao, Wenyi Su, Anastasios Kyrillidis
Conference on Parsimony and Learning, PMLR 280:392-416, 2025.

Abstract

We study a distributed Principal Component Analysis (PCA) framework where each worker targets a distinct eigenvector and refines its solution by updating from intermediate solutions provided by peers deemed as "superior". Drawing intuition from the deflation method in centralized eigenvalue problems, our approach breaks the sequential dependency in the deflation steps and allows asynchronous updates of workers, while incurring only a small communication cost. To our knowledge, a gap in the literature – *the theoretical underpinning of such distributed, dynamic interactions among workers* – has remained unaddressed. This paper offers a theoretical analysis explaining why, how, and when these intermediate, hierarchical updates lead to practical and provable convergence in distributed environments. Despite being a theoretical work, our prototype implementation demonstrates that such a distributed PCA algorithm converges effectively and in scalable way: through experiments, our proposed framework offers comparable performance to EigenGame-$\mu$, the state-of-the-art model-parallel PCA solver.

Cite this Paper

BibTeX
@InProceedings{pmlr-v280-liao25a, title = {Provable Model-Parallel Distributed Principal Component Analysis with Parallel Deflation}, author = {Liao, Fangshuo and Su, Wenyi and Kyrillidis, Anastasios}, booktitle = {Conference on Parsimony and Learning}, pages = {392--416}, year = {2025}, editor = {Chen, Beidi and Liu, Shijia and Pilanci, Mert and Su, Weijie and Sulam, Jeremias and Wang, Yuxiang and Zhu, Zhihui}, volume = {280}, series = {Proceedings of Machine Learning Research}, month = {24--27 Mar}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v280/main/assets/liao25a/liao25a.pdf}, url = {https://proceedings.mlr.press/v280/liao25a.html}, abstract = {We study a distributed Principal Component Analysis (PCA) framework where each worker targets a distinct eigenvector and refines its solution by updating from intermediate solutions provided by peers deemed as "superior". Drawing intuition from the deflation method in centralized eigenvalue problems, our approach breaks the sequential dependency in the deflation steps and allows asynchronous updates of workers, while incurring only a small communication cost. To our knowledge, a gap in the literature – *the theoretical underpinning of such distributed, dynamic interactions among workers* – has remained unaddressed. This paper offers a theoretical analysis explaining why, how, and when these intermediate, hierarchical updates lead to practical and provable convergence in distributed environments. Despite being a theoretical work, our prototype implementation demonstrates that such a distributed PCA algorithm converges effectively and in scalable way: through experiments, our proposed framework offers comparable performance to EigenGame-$\mu$, the state-of-the-art model-parallel PCA solver.} }
Endnote
%0 Conference Paper %T Provable Model-Parallel Distributed Principal Component Analysis with Parallel Deflation %A Fangshuo Liao %A Wenyi Su %A Anastasios Kyrillidis %B Conference on Parsimony and Learning %C Proceedings of Machine Learning Research %D 2025 %E Beidi Chen %E Shijia Liu %E Mert Pilanci %E Weijie Su %E Jeremias Sulam %E Yuxiang Wang %E Zhihui Zhu %F pmlr-v280-liao25a %I PMLR %P 392--416 %U https://proceedings.mlr.press/v280/liao25a.html %V 280 %X We study a distributed Principal Component Analysis (PCA) framework where each worker targets a distinct eigenvector and refines its solution by updating from intermediate solutions provided by peers deemed as "superior". Drawing intuition from the deflation method in centralized eigenvalue problems, our approach breaks the sequential dependency in the deflation steps and allows asynchronous updates of workers, while incurring only a small communication cost. To our knowledge, a gap in the literature – *the theoretical underpinning of such distributed, dynamic interactions among workers* – has remained unaddressed. This paper offers a theoretical analysis explaining why, how, and when these intermediate, hierarchical updates lead to practical and provable convergence in distributed environments. Despite being a theoretical work, our prototype implementation demonstrates that such a distributed PCA algorithm converges effectively and in scalable way: through experiments, our proposed framework offers comparable performance to EigenGame-$\mu$, the state-of-the-art model-parallel PCA solver.
APA
Liao, F., Su, W. & Kyrillidis, A.. (2025). Provable Model-Parallel Distributed Principal Component Analysis with Parallel Deflation. Conference on Parsimony and Learning, in Proceedings of Machine Learning Research 280:392-416 Available from https://proceedings.mlr.press/v280/liao25a.html.

Related Material