Efficient Domain Generalization via Common-Specific Low-Rank Decomposition

Vihari Piratla, Praneeth Netrapalli, Sunita Sarawagi
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:7728-7738, 2020.

Abstract

Domain generalization refers to the task of training a model which generalizes to new domains that are not seen during training. We present CSD (Common Specific Decomposition), for this setting, which jointly learns a common component (which generalizes to new domains) and a domain specific component (which overfits on training domains). The domain specific components are discarded after training and only the common component is retained. The algorithm is extremely simple and involves only modifying the final linear classification layer of any given neural network architecture. We present a principled analysis to understand existing approaches, provide identifiability results of CSD, and study the effect of low-rank on domain generalization. We show that CSD either matches or beats state of the art approaches for domain generalization based on domain erasure, domain perturbed data augmentation, and meta-learning. Further diagnostics on rotated MNIST, where domains are interpretable, confirm the hypothesis that CSD successfully disentangles common and domain specific components and hence leads to better domain generalization; moreover, our code and dataset are publicly available at the following URL: \url{https://github.com/vihari/csd}.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-piratla20a, title = {Efficient Domain Generalization via Common-Specific Low-Rank Decomposition}, author = {Piratla, Vihari and Netrapalli, Praneeth and Sarawagi, Sunita}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {7728--7738}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/piratla20a/piratla20a.pdf}, url = {https://proceedings.mlr.press/v119/piratla20a.html}, abstract = {Domain generalization refers to the task of training a model which generalizes to new domains that are not seen during training. We present CSD (Common Specific Decomposition), for this setting, which jointly learns a common component (which generalizes to new domains) and a domain specific component (which overfits on training domains). The domain specific components are discarded after training and only the common component is retained. The algorithm is extremely simple and involves only modifying the final linear classification layer of any given neural network architecture. We present a principled analysis to understand existing approaches, provide identifiability results of CSD, and study the effect of low-rank on domain generalization. We show that CSD either matches or beats state of the art approaches for domain generalization based on domain erasure, domain perturbed data augmentation, and meta-learning. Further diagnostics on rotated MNIST, where domains are interpretable, confirm the hypothesis that CSD successfully disentangles common and domain specific components and hence leads to better domain generalization; moreover, our code and dataset are publicly available at the following URL: \url{https://github.com/vihari/csd}.} }
Endnote
%0 Conference Paper %T Efficient Domain Generalization via Common-Specific Low-Rank Decomposition %A Vihari Piratla %A Praneeth Netrapalli %A Sunita Sarawagi %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-piratla20a %I PMLR %P 7728--7738 %U https://proceedings.mlr.press/v119/piratla20a.html %V 119 %X Domain generalization refers to the task of training a model which generalizes to new domains that are not seen during training. We present CSD (Common Specific Decomposition), for this setting, which jointly learns a common component (which generalizes to new domains) and a domain specific component (which overfits on training domains). The domain specific components are discarded after training and only the common component is retained. The algorithm is extremely simple and involves only modifying the final linear classification layer of any given neural network architecture. We present a principled analysis to understand existing approaches, provide identifiability results of CSD, and study the effect of low-rank on domain generalization. We show that CSD either matches or beats state of the art approaches for domain generalization based on domain erasure, domain perturbed data augmentation, and meta-learning. Further diagnostics on rotated MNIST, where domains are interpretable, confirm the hypothesis that CSD successfully disentangles common and domain specific components and hence leads to better domain generalization; moreover, our code and dataset are publicly available at the following URL: \url{https://github.com/vihari/csd}.
APA
Piratla, V., Netrapalli, P. & Sarawagi, S.. (2020). Efficient Domain Generalization via Common-Specific Low-Rank Decomposition. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:7728-7738 Available from https://proceedings.mlr.press/v119/piratla20a.html.

Related Material