FedSC: Provable Federated Self-supervised Learning with Spectral Contrastive Objective over Non-i.i.d. Data

Shusen Jing, Anlan Yu, Shuai Zhang, Songyang Zhang
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:22304-22325, 2024.

Abstract

Recent efforts have been made to integrate self-supervised learning (SSL) with the framework of federated learning (FL). One unique challenge of federated self-supervised learning (FedSSL) is that the global objective of FedSSL usually does not equal the weighted sum of local SSL objectives. Consequently, conventional approaches, such as federated averaging (FedAvg), fail to precisely minimize the FedSSL global objective, often resulting in suboptimal performance, especially when data is non-i.i.d.. To fill this gap, we propose a provable FedSSL algorithm, named FedSC, based on the spectral contrastive objective. In FedSC, clients share correlation matrices of data representations in addition to model weights periodically, which enables inter-client contrast of data samples in addition to intra-client contrast and contraction, resulting in improved quality of data representations. Differential privacy (DP) protection is deployed to control the additional privacy leakage on local datasets when correlation matrices are shared. We provide theoretical analysis on convergence and extra privacy leakage, and conduct numerical experiments to justify the effectiveness of our proposed algorithm.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-jing24b, title = {{F}ed{SC}: Provable Federated Self-supervised Learning with Spectral Contrastive Objective over Non-i.i.d. Data}, author = {Jing, Shusen and Yu, Anlan and Zhang, Shuai and Zhang, Songyang}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {22304--22325}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/jing24b/jing24b.pdf}, url = {https://proceedings.mlr.press/v235/jing24b.html}, abstract = {Recent efforts have been made to integrate self-supervised learning (SSL) with the framework of federated learning (FL). One unique challenge of federated self-supervised learning (FedSSL) is that the global objective of FedSSL usually does not equal the weighted sum of local SSL objectives. Consequently, conventional approaches, such as federated averaging (FedAvg), fail to precisely minimize the FedSSL global objective, often resulting in suboptimal performance, especially when data is non-i.i.d.. To fill this gap, we propose a provable FedSSL algorithm, named FedSC, based on the spectral contrastive objective. In FedSC, clients share correlation matrices of data representations in addition to model weights periodically, which enables inter-client contrast of data samples in addition to intra-client contrast and contraction, resulting in improved quality of data representations. Differential privacy (DP) protection is deployed to control the additional privacy leakage on local datasets when correlation matrices are shared. We provide theoretical analysis on convergence and extra privacy leakage, and conduct numerical experiments to justify the effectiveness of our proposed algorithm.} }
Endnote
%0 Conference Paper %T FedSC: Provable Federated Self-supervised Learning with Spectral Contrastive Objective over Non-i.i.d. Data %A Shusen Jing %A Anlan Yu %A Shuai Zhang %A Songyang Zhang %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-jing24b %I PMLR %P 22304--22325 %U https://proceedings.mlr.press/v235/jing24b.html %V 235 %X Recent efforts have been made to integrate self-supervised learning (SSL) with the framework of federated learning (FL). One unique challenge of federated self-supervised learning (FedSSL) is that the global objective of FedSSL usually does not equal the weighted sum of local SSL objectives. Consequently, conventional approaches, such as federated averaging (FedAvg), fail to precisely minimize the FedSSL global objective, often resulting in suboptimal performance, especially when data is non-i.i.d.. To fill this gap, we propose a provable FedSSL algorithm, named FedSC, based on the spectral contrastive objective. In FedSC, clients share correlation matrices of data representations in addition to model weights periodically, which enables inter-client contrast of data samples in addition to intra-client contrast and contraction, resulting in improved quality of data representations. Differential privacy (DP) protection is deployed to control the additional privacy leakage on local datasets when correlation matrices are shared. We provide theoretical analysis on convergence and extra privacy leakage, and conduct numerical experiments to justify the effectiveness of our proposed algorithm.
APA
Jing, S., Yu, A., Zhang, S. & Zhang, S.. (2024). FedSC: Provable Federated Self-supervised Learning with Spectral Contrastive Objective over Non-i.i.d. Data. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:22304-22325 Available from https://proceedings.mlr.press/v235/jing24b.html.

Related Material