Learning Latent Space Models with Angular Constraints

Pengtao Xie, Yuntian Deng, Yi Zhou, Abhimanu Kumar, Yaoliang Yu, James Zou, Eric P. Xing
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:3799-3810, 2017.

Abstract

The large model capacity of latent space models (LSMs) enables them to achieve great performance on various applications, but meanwhile renders LSMs to be prone to overfitting. Several recent studies investigate a new type of regularization approach, which encourages components in LSMs to be diverse, for the sake of alleviating overfitting. While they have shown promising empirical effectiveness, in theory why larger “diversity” results in less overfitting is still unclear. To bridge this gap, we propose a new diversity-promoting approach that is both theoretically analyzable and empirically effective. Specifically, we use near-orthogonality to characterize “diversity” and impose angular constraints (ACs) on the components of LSMs to promote diversity. A generalization error analysis shows that larger diversity results in smaller estimation error and larger approximation error. An efficient ADMM algorithm is developed to solve the constrained LSM problems. Experiments demonstrate that ACs improve generalization performance of LSMs and outperform other diversity-promoting approaches.

Cite this Paper

BibTeX
@InProceedings{pmlr-v70-xie17a, title = {Learning Latent Space Models with Angular Constraints}, author = {Pengtao Xie and Yuntian Deng and Yi Zhou and Abhimanu Kumar and Yaoliang Yu and James Zou and Eric P. Xing}, booktitle = {Proceedings of the 34th International Conference on Machine Learning}, pages = {3799--3810}, year = {2017}, editor = {Precup, Doina and Teh, Yee Whye}, volume = {70}, series = {Proceedings of Machine Learning Research}, month = {06--11 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v70/xie17a/xie17a.pdf}, url = {https://proceedings.mlr.press/v70/xie17a.html}, abstract = {The large model capacity of latent space models (LSMs) enables them to achieve great performance on various applications, but meanwhile renders LSMs to be prone to overfitting. Several recent studies investigate a new type of regularization approach, which encourages components in LSMs to be diverse, for the sake of alleviating overfitting. While they have shown promising empirical effectiveness, in theory why larger “diversity” results in less overfitting is still unclear. To bridge this gap, we propose a new diversity-promoting approach that is both theoretically analyzable and empirically effective. Specifically, we use near-orthogonality to characterize “diversity” and impose angular constraints (ACs) on the components of LSMs to promote diversity. A generalization error analysis shows that larger diversity results in smaller estimation error and larger approximation error. An efficient ADMM algorithm is developed to solve the constrained LSM problems. Experiments demonstrate that ACs improve generalization performance of LSMs and outperform other diversity-promoting approaches.} }
Endnote
%0 Conference Paper %T Learning Latent Space Models with Angular Constraints %A Pengtao Xie %A Yuntian Deng %A Yi Zhou %A Abhimanu Kumar %A Yaoliang Yu %A James Zou %A Eric P. Xing %B Proceedings of the 34th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Doina Precup %E Yee Whye Teh %F pmlr-v70-xie17a %I PMLR %P 3799--3810 %U https://proceedings.mlr.press/v70/xie17a.html %V 70 %X The large model capacity of latent space models (LSMs) enables them to achieve great performance on various applications, but meanwhile renders LSMs to be prone to overfitting. Several recent studies investigate a new type of regularization approach, which encourages components in LSMs to be diverse, for the sake of alleviating overfitting. While they have shown promising empirical effectiveness, in theory why larger “diversity” results in less overfitting is still unclear. To bridge this gap, we propose a new diversity-promoting approach that is both theoretically analyzable and empirically effective. Specifically, we use near-orthogonality to characterize “diversity” and impose angular constraints (ACs) on the components of LSMs to promote diversity. A generalization error analysis shows that larger diversity results in smaller estimation error and larger approximation error. An efficient ADMM algorithm is developed to solve the constrained LSM problems. Experiments demonstrate that ACs improve generalization performance of LSMs and outperform other diversity-promoting approaches.
APA
Xie, P., Deng, Y., Zhou, Y., Kumar, A., Yu, Y., Zou, J. & Xing, E.P.. (2017). Learning Latent Space Models with Angular Constraints. Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research 70:3799-3810 Available from https://proceedings.mlr.press/v70/xie17a.html.

Related Material