Transferability vs. Discriminability: Batch Spectral Penalization for Adversarial Domain Adaptation

Xinyang Chen, Sinan Wang, Mingsheng Long, Jianmin Wang
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:1081-1090, 2019.

Abstract

Adversarial domain adaptation has made remarkable advances in learning transferable representations for knowledge transfer across domains. While adversarial learning strengthens the feature transferability which the community focuses on, its impact on the feature discriminability has not been fully explored. In this paper, a series of experiments based on spectral analysis of the feature representations have been conducted, revealing an unexpected deterioration of the discriminability while learning transferable features adversarially. Our key finding is that the eigenvectors with the largest singular values will dominate the feature transferability. As a consequence, the transferability is enhanced at the expense of over penalization of other eigenvectors that embody rich structures crucial for discriminability. Towards this problem, we present Batch Spectral Penalization (BSP), a general approach to penalizing the largest singular values so that other eigenvectors can be relatively strengthened to boost the feature discriminability. Experiments show that the approach significantly improves upon representative adversarial domain adaptation methods to yield state of the art results.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-chen19i, title = {Transferability vs. Discriminability: Batch Spectral Penalization for Adversarial Domain Adaptation}, author = {Chen, Xinyang and Wang, Sinan and Long, Mingsheng and Wang, Jianmin}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {1081--1090}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/chen19i/chen19i.pdf}, url = {https://proceedings.mlr.press/v97/chen19i.html}, abstract = {Adversarial domain adaptation has made remarkable advances in learning transferable representations for knowledge transfer across domains. While adversarial learning strengthens the feature transferability which the community focuses on, its impact on the feature discriminability has not been fully explored. In this paper, a series of experiments based on spectral analysis of the feature representations have been conducted, revealing an unexpected deterioration of the discriminability while learning transferable features adversarially. Our key finding is that the eigenvectors with the largest singular values will dominate the feature transferability. As a consequence, the transferability is enhanced at the expense of over penalization of other eigenvectors that embody rich structures crucial for discriminability. Towards this problem, we present Batch Spectral Penalization (BSP), a general approach to penalizing the largest singular values so that other eigenvectors can be relatively strengthened to boost the feature discriminability. Experiments show that the approach significantly improves upon representative adversarial domain adaptation methods to yield state of the art results.} }
Endnote
%0 Conference Paper %T Transferability vs. Discriminability: Batch Spectral Penalization for Adversarial Domain Adaptation %A Xinyang Chen %A Sinan Wang %A Mingsheng Long %A Jianmin Wang %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-chen19i %I PMLR %P 1081--1090 %U https://proceedings.mlr.press/v97/chen19i.html %V 97 %X Adversarial domain adaptation has made remarkable advances in learning transferable representations for knowledge transfer across domains. While adversarial learning strengthens the feature transferability which the community focuses on, its impact on the feature discriminability has not been fully explored. In this paper, a series of experiments based on spectral analysis of the feature representations have been conducted, revealing an unexpected deterioration of the discriminability while learning transferable features adversarially. Our key finding is that the eigenvectors with the largest singular values will dominate the feature transferability. As a consequence, the transferability is enhanced at the expense of over penalization of other eigenvectors that embody rich structures crucial for discriminability. Towards this problem, we present Batch Spectral Penalization (BSP), a general approach to penalizing the largest singular values so that other eigenvectors can be relatively strengthened to boost the feature discriminability. Experiments show that the approach significantly improves upon representative adversarial domain adaptation methods to yield state of the art results.
APA
Chen, X., Wang, S., Long, M. & Wang, J.. (2019). Transferability vs. Discriminability: Batch Spectral Penalization for Adversarial Domain Adaptation. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:1081-1090 Available from https://proceedings.mlr.press/v97/chen19i.html.

Related Material