Generalization and Memorization: The Bias Potential Model

Hongkang Yang, Weinan E
Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, PMLR 145:1013-1043, 2022.

Abstract

Models for learning probability distributions such as generative models and density estimators be- have quite differently from models for learning functions. One example is found in the memo- rization phenomenon, namely the ultimate convergence to the empirical distribution, that occurs in generative adversarial networks (GANs). For this reason, the issue of generalization is more subtle than that for supervised learning. For the bias potential model, we show that dimension- independent generalization accuracy is achievable if early stopping is adopted, despite that in the long term, the model either memorizes the samples or diverges.

Cite this Paper

BibTeX
@InProceedings{pmlr-v145-yang22a, title = {Generalization and Memorization: The Bias Potential Model}, author = {Yang, Hongkang and E, Weinan}, booktitle = {Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference}, pages = {1013--1043}, year = {2022}, editor = {Bruna, Joan and Hesthaven, Jan and Zdeborova, Lenka}, volume = {145}, series = {Proceedings of Machine Learning Research}, month = {16--19 Aug}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v145/yang22a/yang22a.pdf}, url = {https://proceedings.mlr.press/v145/yang22a.html}, abstract = {Models for learning probability distributions such as generative models and density estimators be- have quite differently from models for learning functions. One example is found in the memo- rization phenomenon, namely the ultimate convergence to the empirical distribution, that occurs in generative adversarial networks (GANs). For this reason, the issue of generalization is more subtle than that for supervised learning. For the bias potential model, we show that dimension- independent generalization accuracy is achievable if early stopping is adopted, despite that in the long term, the model either memorizes the samples or diverges. } }
Endnote
%0 Conference Paper %T Generalization and Memorization: The Bias Potential Model %A Hongkang Yang %A Weinan E %B Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference %C Proceedings of Machine Learning Research %D 2022 %E Joan Bruna %E Jan Hesthaven %E Lenka Zdeborova %F pmlr-v145-yang22a %I PMLR %P 1013--1043 %U https://proceedings.mlr.press/v145/yang22a.html %V 145 %X Models for learning probability distributions such as generative models and density estimators be- have quite differently from models for learning functions. One example is found in the memo- rization phenomenon, namely the ultimate convergence to the empirical distribution, that occurs in generative adversarial networks (GANs). For this reason, the issue of generalization is more subtle than that for supervised learning. For the bias potential model, we show that dimension- independent generalization accuracy is achievable if early stopping is adopted, despite that in the long term, the model either memorizes the samples or diverges.
APA
Yang, H. & E, W.. (2022). Generalization and Memorization: The Bias Potential Model. Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, in Proceedings of Machine Learning Research 145:1013-1043 Available from https://proceedings.mlr.press/v145/yang22a.html.

Related Material