Feature Grouping as a Stochastic Regularizer for High-Dimensional Structured Data

Sergul Aydore, Bertrand Thirion, Gael Varoquaux
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:385-394, 2019.

Abstract

In many applications where collecting data is expensive, for example neuroscience or medical imaging, the sample size is typically small compared to the feature dimension. These datasets call for intelligent regularization that exploits known structure, such as correlations between the features arising from the measurement device. However, existing structured regularizers need specially crafted solvers, which are difficult to apply to complex models. We propose a new regularizer specifically designed to leverage structure in the data in a way that can be applied efficiently to complex models. Our approach relies on feature grouping, using a fast clustering algorithm inside a stochastic gradient descent loop: given a family of feature groupings that capture feature covariations, we randomly select these groups at each iteration. Experiments on two real-world datasets demonstrate that the proposed approach produces models that generalize better than those trained with conventional regularizers, and also improves convergence speed, and has a linear computational cost.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-aydore19a, title = {Feature Grouping as a Stochastic Regularizer for High-Dimensional Structured Data}, author = {Aydore, Sergul and Thirion, Bertrand and Varoquaux, Gael}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {385--394}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/aydore19a/aydore19a.pdf}, url = {https://proceedings.mlr.press/v97/aydore19a.html}, abstract = {In many applications where collecting data is expensive, for example neuroscience or medical imaging, the sample size is typically small compared to the feature dimension. These datasets call for intelligent regularization that exploits known structure, such as correlations between the features arising from the measurement device. However, existing structured regularizers need specially crafted solvers, which are difficult to apply to complex models. We propose a new regularizer specifically designed to leverage structure in the data in a way that can be applied efficiently to complex models. Our approach relies on feature grouping, using a fast clustering algorithm inside a stochastic gradient descent loop: given a family of feature groupings that capture feature covariations, we randomly select these groups at each iteration. Experiments on two real-world datasets demonstrate that the proposed approach produces models that generalize better than those trained with conventional regularizers, and also improves convergence speed, and has a linear computational cost.} }
Endnote
%0 Conference Paper %T Feature Grouping as a Stochastic Regularizer for High-Dimensional Structured Data %A Sergul Aydore %A Bertrand Thirion %A Gael Varoquaux %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-aydore19a %I PMLR %P 385--394 %U https://proceedings.mlr.press/v97/aydore19a.html %V 97 %X In many applications where collecting data is expensive, for example neuroscience or medical imaging, the sample size is typically small compared to the feature dimension. These datasets call for intelligent regularization that exploits known structure, such as correlations between the features arising from the measurement device. However, existing structured regularizers need specially crafted solvers, which are difficult to apply to complex models. We propose a new regularizer specifically designed to leverage structure in the data in a way that can be applied efficiently to complex models. Our approach relies on feature grouping, using a fast clustering algorithm inside a stochastic gradient descent loop: given a family of feature groupings that capture feature covariations, we randomly select these groups at each iteration. Experiments on two real-world datasets demonstrate that the proposed approach produces models that generalize better than those trained with conventional regularizers, and also improves convergence speed, and has a linear computational cost.
APA
Aydore, S., Thirion, B. & Varoquaux, G.. (2019). Feature Grouping as a Stochastic Regularizer for High-Dimensional Structured Data. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:385-394 Available from https://proceedings.mlr.press/v97/aydore19a.html.

Related Material