Run-Sort-ReRun: Escaping Batch Size Limitations in Sliced Wasserstein Generative Models

Jose Lezama, Wei Chen, Qiang Qiu
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:6275-6285, 2021.

Abstract

When training an implicit generative model, ideally one would like the generator to reproduce all the different modes and subtleties of the target distribution. Naturally, when comparing two empirical distributions, the larger the sample population, the more these statistical nuances can be captured. However, existing objective functions are computationally constrained in the amount of samples they can consider by the memory required to process a batch of samples. In this paper, we build upon recent progress in sliced Wasserstein distances, a family of differentiable metrics for distribution discrepancy based on the Optimal Transport paradigm. We introduce a procedure to train these distances with virtually any batch size, allowing the discrepancy measure to capture richer statistics and better approximating the distance between the underlying continuous distributions. As an example, we demonstrate the matching of the distribution of Inception features with batches of tens of thousands of samples, achieving FID scores that outperform state-of-the-art implicit generative models.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-lezama21a, title = {Run-Sort-ReRun: Escaping Batch Size Limitations in Sliced Wasserstein Generative Models}, author = {Lezama, Jose and Chen, Wei and Qiu, Qiang}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {6275--6285}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/lezama21a/lezama21a.pdf}, url = {https://proceedings.mlr.press/v139/lezama21a.html}, abstract = {When training an implicit generative model, ideally one would like the generator to reproduce all the different modes and subtleties of the target distribution. Naturally, when comparing two empirical distributions, the larger the sample population, the more these statistical nuances can be captured. However, existing objective functions are computationally constrained in the amount of samples they can consider by the memory required to process a batch of samples. In this paper, we build upon recent progress in sliced Wasserstein distances, a family of differentiable metrics for distribution discrepancy based on the Optimal Transport paradigm. We introduce a procedure to train these distances with virtually any batch size, allowing the discrepancy measure to capture richer statistics and better approximating the distance between the underlying continuous distributions. As an example, we demonstrate the matching of the distribution of Inception features with batches of tens of thousands of samples, achieving FID scores that outperform state-of-the-art implicit generative models.} }
Endnote
%0 Conference Paper %T Run-Sort-ReRun: Escaping Batch Size Limitations in Sliced Wasserstein Generative Models %A Jose Lezama %A Wei Chen %A Qiang Qiu %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-lezama21a %I PMLR %P 6275--6285 %U https://proceedings.mlr.press/v139/lezama21a.html %V 139 %X When training an implicit generative model, ideally one would like the generator to reproduce all the different modes and subtleties of the target distribution. Naturally, when comparing two empirical distributions, the larger the sample population, the more these statistical nuances can be captured. However, existing objective functions are computationally constrained in the amount of samples they can consider by the memory required to process a batch of samples. In this paper, we build upon recent progress in sliced Wasserstein distances, a family of differentiable metrics for distribution discrepancy based on the Optimal Transport paradigm. We introduce a procedure to train these distances with virtually any batch size, allowing the discrepancy measure to capture richer statistics and better approximating the distance between the underlying continuous distributions. As an example, we demonstrate the matching of the distribution of Inception features with batches of tens of thousands of samples, achieving FID scores that outperform state-of-the-art implicit generative models.
APA
Lezama, J., Chen, W. & Qiu, Q.. (2021). Run-Sort-ReRun: Escaping Batch Size Limitations in Sliced Wasserstein Generative Models. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:6275-6285 Available from https://proceedings.mlr.press/v139/lezama21a.html.

Related Material