Uncorrelation and Evenness: a New Diversity-Promoting Regularizer

Pengtao Xie, Aarti Singh, Eric P. Xing
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:3811-3820, 2017.

Abstract

Latent space models (LSMs) provide a principled and effective way to extract hidden patterns from observed data. To cope with two challenges in LSMs: (1) how to capture infrequent patterns when pattern frequency is imbalanced and (2) how to reduce model size without sacrificing their expressiveness, several studies have been proposed to “diversify” LSMs, which design regularizers to encourage the components therein to be “diverse”. In light of the limitations of existing approaches, we design a new diversity-promoting regularizer by considering two factors: uncorrelation and evenness, which encourage the components to be uncorrelated and to play equally important roles in modeling data. Formally, this amounts to encouraging the covariance matrix of the components to have more uniform eigenvalues. We apply the regularizer to two LSMs and develop an efficient optimization algorithm. Experiments on healthcare, image and text data demonstrate the effectiveness of the regularizer.

Cite this Paper

BibTeX
@InProceedings{pmlr-v70-xie17b, title = {Uncorrelation and Evenness: a New Diversity-Promoting Regularizer}, author = {Pengtao Xie and Aarti Singh and Eric P. Xing}, booktitle = {Proceedings of the 34th International Conference on Machine Learning}, pages = {3811--3820}, year = {2017}, editor = {Precup, Doina and Teh, Yee Whye}, volume = {70}, series = {Proceedings of Machine Learning Research}, month = {06--11 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v70/xie17b/xie17b.pdf}, url = {https://proceedings.mlr.press/v70/xie17b.html}, abstract = {Latent space models (LSMs) provide a principled and effective way to extract hidden patterns from observed data. To cope with two challenges in LSMs: (1) how to capture infrequent patterns when pattern frequency is imbalanced and (2) how to reduce model size without sacrificing their expressiveness, several studies have been proposed to “diversify” LSMs, which design regularizers to encourage the components therein to be “diverse”. In light of the limitations of existing approaches, we design a new diversity-promoting regularizer by considering two factors: uncorrelation and evenness, which encourage the components to be uncorrelated and to play equally important roles in modeling data. Formally, this amounts to encouraging the covariance matrix of the components to have more uniform eigenvalues. We apply the regularizer to two LSMs and develop an efficient optimization algorithm. Experiments on healthcare, image and text data demonstrate the effectiveness of the regularizer.} }
Endnote
%0 Conference Paper %T Uncorrelation and Evenness: a New Diversity-Promoting Regularizer %A Pengtao Xie %A Aarti Singh %A Eric P. Xing %B Proceedings of the 34th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Doina Precup %E Yee Whye Teh %F pmlr-v70-xie17b %I PMLR %P 3811--3820 %U https://proceedings.mlr.press/v70/xie17b.html %V 70 %X Latent space models (LSMs) provide a principled and effective way to extract hidden patterns from observed data. To cope with two challenges in LSMs: (1) how to capture infrequent patterns when pattern frequency is imbalanced and (2) how to reduce model size without sacrificing their expressiveness, several studies have been proposed to “diversify” LSMs, which design regularizers to encourage the components therein to be “diverse”. In light of the limitations of existing approaches, we design a new diversity-promoting regularizer by considering two factors: uncorrelation and evenness, which encourage the components to be uncorrelated and to play equally important roles in modeling data. Formally, this amounts to encouraging the covariance matrix of the components to have more uniform eigenvalues. We apply the regularizer to two LSMs and develop an efficient optimization algorithm. Experiments on healthcare, image and text data demonstrate the effectiveness of the regularizer.
APA
Xie, P., Singh, A. & Xing, E.P.. (2017). Uncorrelation and Evenness: a New Diversity-Promoting Regularizer. Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research 70:3811-3820 Available from https://proceedings.mlr.press/v70/xie17b.html.

Related Material