On Relativistic f-Divergences

Alexia Jolicoeur-Martineau
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:4931-4939, 2020.

Abstract

We take a more rigorous look at Relativistic Generative Adversarial Networks (RGANs) and prove that the objective function of the discriminator is a statistical divergence for any concave function f with minimal properties. We devise additional variants of relativistic f-divergences. We show that the Wasserstein distance is weaker than f-divergences which are weaker than relativistic f-divergences. Given the good performance of RGANs, this suggests that Wasserstein GAN does not performs well primarily because of the weak metric, but rather because of regularization and the use of a relativistic discriminator. We introduce the minimum-variance unbiased estimator (MVUE) for Relativistic paired GANs (RpGANs; originally called RGANs which could bring confusion) and show that it does not perform better. We show that the estimator of Relativistic average GANs (RaGANs) is asymptotically unbiased and that the finite-sample bias is small; removing this bias does not improve performance.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-jolicoeur-martineau20a, title = {On Relativistic f-Divergences}, author = {Jolicoeur-Martineau, Alexia}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {4931--4939}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/jolicoeur-martineau20a/jolicoeur-martineau20a.pdf}, url = {https://proceedings.mlr.press/v119/jolicoeur-martineau20a.html}, abstract = {We take a more rigorous look at Relativistic Generative Adversarial Networks (RGANs) and prove that the objective function of the discriminator is a statistical divergence for any concave function f with minimal properties. We devise additional variants of relativistic f-divergences. We show that the Wasserstein distance is weaker than f-divergences which are weaker than relativistic f-divergences. Given the good performance of RGANs, this suggests that Wasserstein GAN does not performs well primarily because of the weak metric, but rather because of regularization and the use of a relativistic discriminator. We introduce the minimum-variance unbiased estimator (MVUE) for Relativistic paired GANs (RpGANs; originally called RGANs which could bring confusion) and show that it does not perform better. We show that the estimator of Relativistic average GANs (RaGANs) is asymptotically unbiased and that the finite-sample bias is small; removing this bias does not improve performance.} }
Endnote
%0 Conference Paper %T On Relativistic f-Divergences %A Alexia Jolicoeur-Martineau %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-jolicoeur-martineau20a %I PMLR %P 4931--4939 %U https://proceedings.mlr.press/v119/jolicoeur-martineau20a.html %V 119 %X We take a more rigorous look at Relativistic Generative Adversarial Networks (RGANs) and prove that the objective function of the discriminator is a statistical divergence for any concave function f with minimal properties. We devise additional variants of relativistic f-divergences. We show that the Wasserstein distance is weaker than f-divergences which are weaker than relativistic f-divergences. Given the good performance of RGANs, this suggests that Wasserstein GAN does not performs well primarily because of the weak metric, but rather because of regularization and the use of a relativistic discriminator. We introduce the minimum-variance unbiased estimator (MVUE) for Relativistic paired GANs (RpGANs; originally called RGANs which could bring confusion) and show that it does not perform better. We show that the estimator of Relativistic average GANs (RaGANs) is asymptotically unbiased and that the finite-sample bias is small; removing this bias does not improve performance.
APA
Jolicoeur-Martineau, A.. (2020). On Relativistic f-Divergences. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:4931-4939 Available from https://proceedings.mlr.press/v119/jolicoeur-martineau20a.html.

Related Material