Empirical Study of the Benefits of Overparameterization in Learning Latent Variable Models

Rares-Darius Buhai, Yoni Halpern, Yoon Kim, Andrej Risteski, David Sontag
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:1211-1219, 2020.

Abstract

One of the most surprising and exciting discoveries in supervised learning was the benefit of overparameterization (i.e. training a very large model) to improving the optimization landscape of a problem, with minimal effect on statistical performance (i.e. generalization). In contrast, unsupervised settings have been under-explored, despite the fact that it was observed that overparameterization can be helpful as early as Dasgupta & Schulman (2007). We perform an empirical study of different aspects of overparameterization in unsupervised learning of latent variable models via synthetic and semi-synthetic experiments. We discuss benefits to different metrics of success (recovering the parameters of the ground-truth model, held-out log-likelihood), sensitivity to variations of the training algorithm, and behavior as the amount of overparameterization increases. We find that across a variety of models (noisy-OR networks, sparse coding, probabilistic context-free grammars) and training algorithms (variational inference, alternating minimization, expectation-maximization), overparameterization can significantly increase the number of ground truth latent variables recovered.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-buhai20a, title = {Empirical Study of the Benefits of Overparameterization in Learning Latent Variable Models}, author = {Buhai, Rares-Darius and Halpern, Yoni and Kim, Yoon and Risteski, Andrej and Sontag, David}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {1211--1219}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/buhai20a/buhai20a.pdf}, url = {https://proceedings.mlr.press/v119/buhai20a.html}, abstract = {One of the most surprising and exciting discoveries in supervised learning was the benefit of overparameterization (i.e. training a very large model) to improving the optimization landscape of a problem, with minimal effect on statistical performance (i.e. generalization). In contrast, unsupervised settings have been under-explored, despite the fact that it was observed that overparameterization can be helpful as early as Dasgupta & Schulman (2007). We perform an empirical study of different aspects of overparameterization in unsupervised learning of latent variable models via synthetic and semi-synthetic experiments. We discuss benefits to different metrics of success (recovering the parameters of the ground-truth model, held-out log-likelihood), sensitivity to variations of the training algorithm, and behavior as the amount of overparameterization increases. We find that across a variety of models (noisy-OR networks, sparse coding, probabilistic context-free grammars) and training algorithms (variational inference, alternating minimization, expectation-maximization), overparameterization can significantly increase the number of ground truth latent variables recovered.} }
Endnote
%0 Conference Paper %T Empirical Study of the Benefits of Overparameterization in Learning Latent Variable Models %A Rares-Darius Buhai %A Yoni Halpern %A Yoon Kim %A Andrej Risteski %A David Sontag %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-buhai20a %I PMLR %P 1211--1219 %U https://proceedings.mlr.press/v119/buhai20a.html %V 119 %X One of the most surprising and exciting discoveries in supervised learning was the benefit of overparameterization (i.e. training a very large model) to improving the optimization landscape of a problem, with minimal effect on statistical performance (i.e. generalization). In contrast, unsupervised settings have been under-explored, despite the fact that it was observed that overparameterization can be helpful as early as Dasgupta & Schulman (2007). We perform an empirical study of different aspects of overparameterization in unsupervised learning of latent variable models via synthetic and semi-synthetic experiments. We discuss benefits to different metrics of success (recovering the parameters of the ground-truth model, held-out log-likelihood), sensitivity to variations of the training algorithm, and behavior as the amount of overparameterization increases. We find that across a variety of models (noisy-OR networks, sparse coding, probabilistic context-free grammars) and training algorithms (variational inference, alternating minimization, expectation-maximization), overparameterization can significantly increase the number of ground truth latent variables recovered.
APA
Buhai, R., Halpern, Y., Kim, Y., Risteski, A. & Sontag, D.. (2020). Empirical Study of the Benefits of Overparameterization in Learning Latent Variable Models. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:1211-1219 Available from https://proceedings.mlr.press/v119/buhai20a.html.

Related Material