Low-Rank Approximation of Structural Redundancy for Self-Supervised Learning

Kang Du, Yu Xiang
Proceedings of the Third Conference on Causal Learning and Reasoning, PMLR 236:1008-1032, 2024.

Abstract

We study the data-generating mechanism for reconstructive SSL to shed light on its effectiveness. With an infinite amount of labeled samples, we provide a sufficient and necessary condition for perfect linear approximation. The condition reveals a full-rank component that preserves the label classes of $Y$, along with a redundant component. Motivated by the condition, we propose to approximate the redundant component by a low-rank factorization and measure the approximation quality by introducing a new measure, $\epsilon_s$, parameterized by the rank of factorization $s$. We incorporate $\epsilon_s$ into the excess risk analysis under both linear regression and ridge regression settings, where the latter regularization approach is to handle scenarios when the dimension of the learned features is much larger than the number of labeled samples $n$ for downstream tasks. We design two stylized experiments to compare SSL with supervised learning (SL) under different settings to support our theoretical findings.

Cite this Paper

BibTeX
@InProceedings{pmlr-v236-du24a, title = {Low-Rank Approximation of Structural Redundancy for Self-Supervised Learning}, author = {Du, Kang and Xiang, Yu}, booktitle = {Proceedings of the Third Conference on Causal Learning and Reasoning}, pages = {1008--1032}, year = {2024}, editor = {Locatello, Francesco and Didelez, Vanessa}, volume = {236}, series = {Proceedings of Machine Learning Research}, month = {01--03 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v236/du24a/du24a.pdf}, url = {https://proceedings.mlr.press/v236/du24a.html}, abstract = {We study the data-generating mechanism for reconstructive SSL to shed light on its effectiveness. With an infinite amount of labeled samples, we provide a sufficient and necessary condition for perfect linear approximation. The condition reveals a full-rank component that preserves the label classes of $Y$, along with a redundant component. Motivated by the condition, we propose to approximate the redundant component by a low-rank factorization and measure the approximation quality by introducing a new measure, $\epsilon_s$, parameterized by the rank of factorization $s$. We incorporate $\epsilon_s$ into the excess risk analysis under both linear regression and ridge regression settings, where the latter regularization approach is to handle scenarios when the dimension of the learned features is much larger than the number of labeled samples $n$ for downstream tasks. We design two stylized experiments to compare SSL with supervised learning (SL) under different settings to support our theoretical findings.} }
Endnote
%0 Conference Paper %T Low-Rank Approximation of Structural Redundancy for Self-Supervised Learning %A Kang Du %A Yu Xiang %B Proceedings of the Third Conference on Causal Learning and Reasoning %C Proceedings of Machine Learning Research %D 2024 %E Francesco Locatello %E Vanessa Didelez %F pmlr-v236-du24a %I PMLR %P 1008--1032 %U https://proceedings.mlr.press/v236/du24a.html %V 236 %X We study the data-generating mechanism for reconstructive SSL to shed light on its effectiveness. With an infinite amount of labeled samples, we provide a sufficient and necessary condition for perfect linear approximation. The condition reveals a full-rank component that preserves the label classes of $Y$, along with a redundant component. Motivated by the condition, we propose to approximate the redundant component by a low-rank factorization and measure the approximation quality by introducing a new measure, $\epsilon_s$, parameterized by the rank of factorization $s$. We incorporate $\epsilon_s$ into the excess risk analysis under both linear regression and ridge regression settings, where the latter regularization approach is to handle scenarios when the dimension of the learned features is much larger than the number of labeled samples $n$ for downstream tasks. We design two stylized experiments to compare SSL with supervised learning (SL) under different settings to support our theoretical findings.
APA
Du, K. & Xiang, Y.. (2024). Low-Rank Approximation of Structural Redundancy for Self-Supervised Learning. Proceedings of the Third Conference on Causal Learning and Reasoning, in Proceedings of Machine Learning Research 236:1008-1032 Available from https://proceedings.mlr.press/v236/du24a.html.

Related Material