Why Regularized Auto-Encoders learn Sparse Representation?

Devansh Arpit, Yingbo Zhou, Hung Ngo, Venu Govindaraju
Proceedings of The 33rd International Conference on Machine Learning, PMLR 48:136-144, 2016.

Abstract

Sparse distributed representation is the key to learning useful features in deep learning algorithms, because not only it is an efficient mode of data representation, but also – more importantly – it captures the generation process of most real world data. While a number of regularized auto-encoders (AE) enforce sparsity explicitly in their learned representation and others don’t, there has been little formal analysis on what encourages sparsity in these models in general. Our objective is to formally study this general problem for regularized auto-encoders. We provide sufficient conditions on both regularization and activation functions that encourage sparsity. We show that multiple popular models (de-noising and contractive auto encoders, e.g.) and activations (rectified linear and sigmoid, e.g.) satisfy these conditions; thus, our conditions help explain sparsity in their learned representation. Thus our theoretical and empirical analysis together shed light on the properties of regularization/activation that are conductive to sparsity and unify a number of existing auto-encoder models and activation functions under the same analytical framework.

Cite this Paper

BibTeX
@InProceedings{pmlr-v48-arpita16, title = {Why Regularized Auto-Encoders learn Sparse Representation?}, author = {Arpit, Devansh and Zhou, Yingbo and Ngo, Hung and Govindaraju, Venu}, booktitle = {Proceedings of The 33rd International Conference on Machine Learning}, pages = {136--144}, year = {2016}, editor = {Balcan, Maria Florina and Weinberger, Kilian Q.}, volume = {48}, series = {Proceedings of Machine Learning Research}, address = {New York, New York, USA}, month = {20--22 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v48/arpita16.pdf}, url = {https://proceedings.mlr.press/v48/arpita16.html}, abstract = {Sparse distributed representation is the key to learning useful features in deep learning algorithms, because not only it is an efficient mode of data representation, but also – more importantly – it captures the generation process of most real world data. While a number of regularized auto-encoders (AE) enforce sparsity explicitly in their learned representation and others don’t, there has been little formal analysis on what encourages sparsity in these models in general. Our objective is to formally study this general problem for regularized auto-encoders. We provide sufficient conditions on both regularization and activation functions that encourage sparsity. We show that multiple popular models (de-noising and contractive auto encoders, e.g.) and activations (rectified linear and sigmoid, e.g.) satisfy these conditions; thus, our conditions help explain sparsity in their learned representation. Thus our theoretical and empirical analysis together shed light on the properties of regularization/activation that are conductive to sparsity and unify a number of existing auto-encoder models and activation functions under the same analytical framework.} }
Endnote
%0 Conference Paper %T Why Regularized Auto-Encoders learn Sparse Representation? %A Devansh Arpit %A Yingbo Zhou %A Hung Ngo %A Venu Govindaraju %B Proceedings of The 33rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2016 %E Maria Florina Balcan %E Kilian Q. Weinberger %F pmlr-v48-arpita16 %I PMLR %P 136--144 %U https://proceedings.mlr.press/v48/arpita16.html %V 48 %X Sparse distributed representation is the key to learning useful features in deep learning algorithms, because not only it is an efficient mode of data representation, but also – more importantly – it captures the generation process of most real world data. While a number of regularized auto-encoders (AE) enforce sparsity explicitly in their learned representation and others don’t, there has been little formal analysis on what encourages sparsity in these models in general. Our objective is to formally study this general problem for regularized auto-encoders. We provide sufficient conditions on both regularization and activation functions that encourage sparsity. We show that multiple popular models (de-noising and contractive auto encoders, e.g.) and activations (rectified linear and sigmoid, e.g.) satisfy these conditions; thus, our conditions help explain sparsity in their learned representation. Thus our theoretical and empirical analysis together shed light on the properties of regularization/activation that are conductive to sparsity and unify a number of existing auto-encoder models and activation functions under the same analytical framework.
RIS
TY - CPAPER TI - Why Regularized Auto-Encoders learn Sparse Representation? AU - Devansh Arpit AU - Yingbo Zhou AU - Hung Ngo AU - Venu Govindaraju BT - Proceedings of The 33rd International Conference on Machine Learning DA - 2016/06/11 ED - Maria Florina Balcan ED - Kilian Q. Weinberger ID - pmlr-v48-arpita16 PB - PMLR DP - Proceedings of Machine Learning Research VL - 48 SP - 136 EP - 144 L1 - http://proceedings.mlr.press/v48/arpita16.pdf UR - https://proceedings.mlr.press/v48/arpita16.html AB - Sparse distributed representation is the key to learning useful features in deep learning algorithms, because not only it is an efficient mode of data representation, but also – more importantly – it captures the generation process of most real world data. While a number of regularized auto-encoders (AE) enforce sparsity explicitly in their learned representation and others don’t, there has been little formal analysis on what encourages sparsity in these models in general. Our objective is to formally study this general problem for regularized auto-encoders. We provide sufficient conditions on both regularization and activation functions that encourage sparsity. We show that multiple popular models (de-noising and contractive auto encoders, e.g.) and activations (rectified linear and sigmoid, e.g.) satisfy these conditions; thus, our conditions help explain sparsity in their learned representation. Thus our theoretical and empirical analysis together shed light on the properties of regularization/activation that are conductive to sparsity and unify a number of existing auto-encoder models and activation functions under the same analytical framework. ER -
APA
Arpit, D., Zhou, Y., Ngo, H. & Govindaraju, V.. (2016). Why Regularized Auto-Encoders learn Sparse Representation?. Proceedings of The 33rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 48:136-144 Available from https://proceedings.mlr.press/v48/arpita16.html.

Related Material