Small-GAN: Speeding up GAN Training using Core-Sets

Samarth Sinha, Han Zhang, Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, Augustus Odena
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:9005-9015, 2020.

Abstract

Recent work suggests that Generative Adversarial Networks (GANs) benefit disproportionately from large mini-batch sizes. This finding is interesting but also discouraging – large batch sizes are slow and expensive to emulate on conventional hardware. Thus, it would be nice if there were some trick by which we could generate batches that were effectively big though small in practice. In this work, we propose such a trick, inspired by the use of Coreset-selection in active learning. When training a GAN, we draw a large batch of samples from the prior and then compress that batch using Coreset-selection. To create effectively large batches of real images, we create a cached dataset of Inception activations of each training image, randomly project them down to a smaller dimension, and then use Coreset-selection on those projected embeddings at training time. We conduct experiments showing that this technique substantially reduces training time and memory usage for modern GAN variants, that it reduces the fraction of dropped modes in a synthetic dataset, and that it helps us use GANs to reach a new state of the art in anomaly detection.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-sinha20b, title = {Small-{GAN}: Speeding up {GAN} Training using Core-Sets}, author = {Sinha, Samarth and Zhang, Han and Goyal, Anirudh and Bengio, Yoshua and Larochelle, Hugo and Odena, Augustus}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {9005--9015}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/sinha20b/sinha20b.pdf}, url = {https://proceedings.mlr.press/v119/sinha20b.html}, abstract = {Recent work suggests that Generative Adversarial Networks (GANs) benefit disproportionately from large mini-batch sizes. This finding is interesting but also discouraging – large batch sizes are slow and expensive to emulate on conventional hardware. Thus, it would be nice if there were some trick by which we could generate batches that were effectively big though small in practice. In this work, we propose such a trick, inspired by the use of Coreset-selection in active learning. When training a GAN, we draw a large batch of samples from the prior and then compress that batch using Coreset-selection. To create effectively large batches of real images, we create a cached dataset of Inception activations of each training image, randomly project them down to a smaller dimension, and then use Coreset-selection on those projected embeddings at training time. We conduct experiments showing that this technique substantially reduces training time and memory usage for modern GAN variants, that it reduces the fraction of dropped modes in a synthetic dataset, and that it helps us use GANs to reach a new state of the art in anomaly detection.} }
Endnote
%0 Conference Paper %T Small-GAN: Speeding up GAN Training using Core-Sets %A Samarth Sinha %A Han Zhang %A Anirudh Goyal %A Yoshua Bengio %A Hugo Larochelle %A Augustus Odena %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-sinha20b %I PMLR %P 9005--9015 %U https://proceedings.mlr.press/v119/sinha20b.html %V 119 %X Recent work suggests that Generative Adversarial Networks (GANs) benefit disproportionately from large mini-batch sizes. This finding is interesting but also discouraging – large batch sizes are slow and expensive to emulate on conventional hardware. Thus, it would be nice if there were some trick by which we could generate batches that were effectively big though small in practice. In this work, we propose such a trick, inspired by the use of Coreset-selection in active learning. When training a GAN, we draw a large batch of samples from the prior and then compress that batch using Coreset-selection. To create effectively large batches of real images, we create a cached dataset of Inception activations of each training image, randomly project them down to a smaller dimension, and then use Coreset-selection on those projected embeddings at training time. We conduct experiments showing that this technique substantially reduces training time and memory usage for modern GAN variants, that it reduces the fraction of dropped modes in a synthetic dataset, and that it helps us use GANs to reach a new state of the art in anomaly detection.
APA
Sinha, S., Zhang, H., Goyal, A., Bengio, Y., Larochelle, H. & Odena, A.. (2020). Small-GAN: Speeding up GAN Training using Core-Sets. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:9005-9015 Available from https://proceedings.mlr.press/v119/sinha20b.html.

Related Material