Replaying with Realistic Latent Vectors in Generative Continual Learning

In generative continual learning for GANs, replay-based methods mitigate forgetting of past knowledge by retraining either synthesised images from the previous generator (generative replay) or real images from the memory buffer (memory replay) during training each new task. However, despite its strength at memory efficiency, generative replay often fails to produce realistic images, due to the gap between synthetic and real images especially in challenging datasets. Although memory replay can address this issue by storing real images, it still suffers from the practical limit of memory space, gradually deteriorating the quality of generated images. In this paper, we propose a novel mixed replay method, called Realistic Latent Vector Fitting (RactoFit), which not only effectively resolves the drawbacks of generative and memory replay but also combine their strengths, memory utilisation yet realism preservation. To efficiently utilize the memory space, our proposal is to maintain realistic latent vectors corresponding to previous real images, rather than storing those images themselves. In addition to replaying with these stored latent vectors, we also design a new optimization technique for latent vectors that can generate more realistic images from the previous generator, not just taking random vectors as in generative replay. Through our experimental analysis, we demonstrate that our method outperforms the baseline methods of continual learning for GANs in terms of the standard metrics for generative models, along with qualitative results clearly showing the highest level of realism of our generated images.