Continuously Parameterized Mixture Models

Christopher M Bender, Yifeng Shi, Marc Niethammer, Junier Oliva
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:2050-2062, 2023.

Abstract

Mixture models are universal approximators of smooth densities but are difficult to utilize in complicated datasets due to restrictions on typically available modes and challenges with initialiations. We show that by continuously parameterizing a mixture of factor analyzers using a learned ordinary differential equation, we can improve the fit of mixture models over direct methods. Once trained, the mixture components can be extracted and the neural ODE can be discarded, leaving us with an effective, but low-resource model. We additionally explore the use of a training curriculum from an easy-to-model latent space extracted from a normalizing flow to the more complex input space and show that the smooth curriculum helps to stabilize and improve results with and without the continuous parameterization. Finally, we introduce a hierarchical version of the model to enable more flexible, robust classification and clustering, and show substantial improvements against traditional parameterizations of GMMs.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-bender23a, title = {Continuously Parameterized Mixture Models}, author = {Bender, Christopher M and Shi, Yifeng and Niethammer, Marc and Oliva, Junier}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {2050--2062}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/bender23a/bender23a.pdf}, url = {https://proceedings.mlr.press/v202/bender23a.html}, abstract = {Mixture models are universal approximators of smooth densities but are difficult to utilize in complicated datasets due to restrictions on typically available modes and challenges with initialiations. We show that by continuously parameterizing a mixture of factor analyzers using a learned ordinary differential equation, we can improve the fit of mixture models over direct methods. Once trained, the mixture components can be extracted and the neural ODE can be discarded, leaving us with an effective, but low-resource model. We additionally explore the use of a training curriculum from an easy-to-model latent space extracted from a normalizing flow to the more complex input space and show that the smooth curriculum helps to stabilize and improve results with and without the continuous parameterization. Finally, we introduce a hierarchical version of the model to enable more flexible, robust classification and clustering, and show substantial improvements against traditional parameterizations of GMMs.} }
Endnote
%0 Conference Paper %T Continuously Parameterized Mixture Models %A Christopher M Bender %A Yifeng Shi %A Marc Niethammer %A Junier Oliva %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-bender23a %I PMLR %P 2050--2062 %U https://proceedings.mlr.press/v202/bender23a.html %V 202 %X Mixture models are universal approximators of smooth densities but are difficult to utilize in complicated datasets due to restrictions on typically available modes and challenges with initialiations. We show that by continuously parameterizing a mixture of factor analyzers using a learned ordinary differential equation, we can improve the fit of mixture models over direct methods. Once trained, the mixture components can be extracted and the neural ODE can be discarded, leaving us with an effective, but low-resource model. We additionally explore the use of a training curriculum from an easy-to-model latent space extracted from a normalizing flow to the more complex input space and show that the smooth curriculum helps to stabilize and improve results with and without the continuous parameterization. Finally, we introduce a hierarchical version of the model to enable more flexible, robust classification and clustering, and show substantial improvements against traditional parameterizations of GMMs.
APA
Bender, C.M., Shi, Y., Niethammer, M. & Oliva, J.. (2023). Continuously Parameterized Mixture Models. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:2050-2062 Available from https://proceedings.mlr.press/v202/bender23a.html.

Related Material