Algorithm-Dependent Bounds for Representation Learning of Multi-Source Domain Adaptation

Qi Chen, Mario Marchand
Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, PMLR 206:10368-10394, 2023.

Abstract

We use information-theoretic tools to derive a novel analysis of Multi-source Domain Adaptation (MDA) from the representation learning perspective. Concretely, we study joint distribution alignment for supervised MDA with few target labels and unsupervised MDA with pseudo labels, where the latter is relatively hard and less commonly studied. We further provide algorithm-dependent generalization bounds for these two settings, where the generalization is characterized by the mutual information between the parameters and the data. Then we propose a novel deep MDA algorithm, implicitly addressing the target shift through joint alignment. Finally, the mutual information bounds are extended to this algorithm providing a non-vacuous gradient-norm estimation. The proposed algorithm has comparable performance to the state-of-the-art on target-shifted MDA benchmark with improved memory efficiency.

Cite this Paper

BibTeX
@InProceedings{pmlr-v206-chen23h, title = {Algorithm-Dependent Bounds for Representation Learning of Multi-Source Domain Adaptation}, author = {Chen, Qi and Marchand, Mario}, booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics}, pages = {10368--10394}, year = {2023}, editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem}, volume = {206}, series = {Proceedings of Machine Learning Research}, month = {25--27 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v206/chen23h/chen23h.pdf}, url = {https://proceedings.mlr.press/v206/chen23h.html}, abstract = {We use information-theoretic tools to derive a novel analysis of Multi-source Domain Adaptation (MDA) from the representation learning perspective. Concretely, we study joint distribution alignment for supervised MDA with few target labels and unsupervised MDA with pseudo labels, where the latter is relatively hard and less commonly studied. We further provide algorithm-dependent generalization bounds for these two settings, where the generalization is characterized by the mutual information between the parameters and the data. Then we propose a novel deep MDA algorithm, implicitly addressing the target shift through joint alignment. Finally, the mutual information bounds are extended to this algorithm providing a non-vacuous gradient-norm estimation. The proposed algorithm has comparable performance to the state-of-the-art on target-shifted MDA benchmark with improved memory efficiency.} }
Endnote
%0 Conference Paper %T Algorithm-Dependent Bounds for Representation Learning of Multi-Source Domain Adaptation %A Qi Chen %A Mario Marchand %B Proceedings of The 26th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2023 %E Francisco Ruiz %E Jennifer Dy %E Jan-Willem van de Meent %F pmlr-v206-chen23h %I PMLR %P 10368--10394 %U https://proceedings.mlr.press/v206/chen23h.html %V 206 %X We use information-theoretic tools to derive a novel analysis of Multi-source Domain Adaptation (MDA) from the representation learning perspective. Concretely, we study joint distribution alignment for supervised MDA with few target labels and unsupervised MDA with pseudo labels, where the latter is relatively hard and less commonly studied. We further provide algorithm-dependent generalization bounds for these two settings, where the generalization is characterized by the mutual information between the parameters and the data. Then we propose a novel deep MDA algorithm, implicitly addressing the target shift through joint alignment. Finally, the mutual information bounds are extended to this algorithm providing a non-vacuous gradient-norm estimation. The proposed algorithm has comparable performance to the state-of-the-art on target-shifted MDA benchmark with improved memory efficiency.
APA
Chen, Q. & Marchand, M.. (2023). Algorithm-Dependent Bounds for Representation Learning of Multi-Source Domain Adaptation. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 206:10368-10394 Available from https://proceedings.mlr.press/v206/chen23h.html.

Related Material