Maximally Informative Hierarchical Representations of High-Dimensional Data

Greg Ver Steeg, Aram Galstyan
Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, PMLR 38:1004-1012, 2015.

Abstract

We consider a set of probabilistic functions of some input variables as a representation of the inputs. We present bounds on how informative a representation is about input data. We extend these bounds to hierarchical representations so that we can quantify the contribution of each layer towards capturing the information in the original data. The special form of these bounds leads to a simple, bottom-up optimization procedure to construct hierarchical representations that are also maximally informative about the data. This optimization has linear computational complexity and constant sample complexity in the number of variables. These results establish a new approach to unsupervised learning of deep representations that is both principled and practical. We demonstrate the usefulness of the approach on both synthetic and real-world data.

Cite this Paper

BibTeX
@InProceedings{pmlr-v38-versteeg15, title = {{Maximally Informative Hierarchical Representations of High-Dimensional Data}}, author = {Ver Steeg, Greg and Galstyan, Aram}, booktitle = {Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics}, pages = {1004--1012}, year = {2015}, editor = {Lebanon, Guy and Vishwanathan, S. V. N.}, volume = {38}, series = {Proceedings of Machine Learning Research}, address = {San Diego, California, USA}, month = {09--12 May}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v38/versteeg15.pdf}, url = {https://proceedings.mlr.press/v38/versteeg15.html}, abstract = {We consider a set of probabilistic functions of some input variables as a representation of the inputs. We present bounds on how informative a representation is about input data. We extend these bounds to hierarchical representations so that we can quantify the contribution of each layer towards capturing the information in the original data. The special form of these bounds leads to a simple, bottom-up optimization procedure to construct hierarchical representations that are also maximally informative about the data. This optimization has linear computational complexity and constant sample complexity in the number of variables. These results establish a new approach to unsupervised learning of deep representations that is both principled and practical. We demonstrate the usefulness of the approach on both synthetic and real-world data.} }
Endnote
%0 Conference Paper %T Maximally Informative Hierarchical Representations of High-Dimensional Data %A Greg Ver Steeg %A Aram Galstyan %B Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2015 %E Guy Lebanon %E S. V. N. Vishwanathan %F pmlr-v38-versteeg15 %I PMLR %P 1004--1012 %U https://proceedings.mlr.press/v38/versteeg15.html %V 38 %X We consider a set of probabilistic functions of some input variables as a representation of the inputs. We present bounds on how informative a representation is about input data. We extend these bounds to hierarchical representations so that we can quantify the contribution of each layer towards capturing the information in the original data. The special form of these bounds leads to a simple, bottom-up optimization procedure to construct hierarchical representations that are also maximally informative about the data. This optimization has linear computational complexity and constant sample complexity in the number of variables. These results establish a new approach to unsupervised learning of deep representations that is both principled and practical. We demonstrate the usefulness of the approach on both synthetic and real-world data.
RIS
TY - CPAPER TI - Maximally Informative Hierarchical Representations of High-Dimensional Data AU - Greg Ver Steeg AU - Aram Galstyan BT - Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics DA - 2015/02/21 ED - Guy Lebanon ED - S. V. N. Vishwanathan ID - pmlr-v38-versteeg15 PB - PMLR DP - Proceedings of Machine Learning Research VL - 38 SP - 1004 EP - 1012 L1 - http://proceedings.mlr.press/v38/versteeg15.pdf UR - https://proceedings.mlr.press/v38/versteeg15.html AB - We consider a set of probabilistic functions of some input variables as a representation of the inputs. We present bounds on how informative a representation is about input data. We extend these bounds to hierarchical representations so that we can quantify the contribution of each layer towards capturing the information in the original data. The special form of these bounds leads to a simple, bottom-up optimization procedure to construct hierarchical representations that are also maximally informative about the data. This optimization has linear computational complexity and constant sample complexity in the number of variables. These results establish a new approach to unsupervised learning of deep representations that is both principled and practical. We demonstrate the usefulness of the approach on both synthetic and real-world data. ER -
APA
Ver Steeg, G. & Galstyan, A.. (2015). Maximally Informative Hierarchical Representations of High-Dimensional Data. Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 38:1004-1012 Available from https://proceedings.mlr.press/v38/versteeg15.html.

Related Material