Understanding and Stabilizing GANs’ Training Dynamics Using Control Theory

Kun Xu, Chongxuan Li, Jun Zhu, Bo Zhang
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:10566-10575, 2020.

Abstract

Generative adversarial networks (GANs) are effective in generating realistic images but the training is often unstable. There are existing efforts that model the training dynamics of GANs in the parameter space but the analysis cannot directly motivate practically effective stabilizing methods. To this end, we present a conceptually novel perspective from control theory to directly model the dynamics of GANs in the frequency domain and provide simple yet effective methods to stabilize GAN’s training. We first analyze the training dynamic of a prototypical Dirac GAN and adopt the widely-used closed-loop control (CLC) to improve its stability. We then extend CLC to stabilize the training dynamic of normal GANs, which can be implemented as an L2 regularizer on the output of the discriminator. Empirical results show that our method can effectively stabilize the training and obtain state-of-the-art performance on data generation tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-xu20d, title = {Understanding and Stabilizing {GAN}s’ Training Dynamics Using Control Theory}, author = {Xu, Kun and Li, Chongxuan and Zhu, Jun and Zhang, Bo}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {10566--10575}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/xu20d/xu20d.pdf}, url = {https://proceedings.mlr.press/v119/xu20d.html}, abstract = {Generative adversarial networks (GANs) are effective in generating realistic images but the training is often unstable. There are existing efforts that model the training dynamics of GANs in the parameter space but the analysis cannot directly motivate practically effective stabilizing methods. To this end, we present a conceptually novel perspective from control theory to directly model the dynamics of GANs in the frequency domain and provide simple yet effective methods to stabilize GAN’s training. We first analyze the training dynamic of a prototypical Dirac GAN and adopt the widely-used closed-loop control (CLC) to improve its stability. We then extend CLC to stabilize the training dynamic of normal GANs, which can be implemented as an L2 regularizer on the output of the discriminator. Empirical results show that our method can effectively stabilize the training and obtain state-of-the-art performance on data generation tasks.} }
Endnote
%0 Conference Paper %T Understanding and Stabilizing GANs’ Training Dynamics Using Control Theory %A Kun Xu %A Chongxuan Li %A Jun Zhu %A Bo Zhang %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-xu20d %I PMLR %P 10566--10575 %U https://proceedings.mlr.press/v119/xu20d.html %V 119 %X Generative adversarial networks (GANs) are effective in generating realistic images but the training is often unstable. There are existing efforts that model the training dynamics of GANs in the parameter space but the analysis cannot directly motivate practically effective stabilizing methods. To this end, we present a conceptually novel perspective from control theory to directly model the dynamics of GANs in the frequency domain and provide simple yet effective methods to stabilize GAN’s training. We first analyze the training dynamic of a prototypical Dirac GAN and adopt the widely-used closed-loop control (CLC) to improve its stability. We then extend CLC to stabilize the training dynamic of normal GANs, which can be implemented as an L2 regularizer on the output of the discriminator. Empirical results show that our method can effectively stabilize the training and obtain state-of-the-art performance on data generation tasks.
APA
Xu, K., Li, C., Zhu, J. & Zhang, B.. (2020). Understanding and Stabilizing GANs’ Training Dynamics Using Control Theory. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:10566-10575 Available from https://proceedings.mlr.press/v119/xu20d.html.

Related Material