Holographic Feature Representations of Deep Networks

Martin A. Zinkevich, Alex Davies, Dale Schuurmans
Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence, PMLR R15:401-410, 2017.

Abstract

It is often asserted that deep networks learn “features”, traditionally expressed by the activations of intermediate nodes. We explore an alternative concept by defining features as partial derivatives of model output with respect to model parameters—extending a simple yet powerful idea from generalized linear models. The resulting features are not equivalent to node activations, and we show that they can induce a holographic representation of the complete model: the network’s output on given data can be exactly replicated by a simple linear model over such features extracted from any ordered cut. We demonstrate useful advantages for this feature representation over standard representations based on node activations.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR15-zinkevich17a, title = {Holographic Feature Representations of Deep Networks}, author = {Zinkevich, Martin A. and Davies, Alex and Schuurmans, Dale}, booktitle = {Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence}, pages = {401--410}, year = {2017}, editor = {Elidan, Gal and Kersting, Kristian}, volume = {R15}, series = {Proceedings of Machine Learning Research}, month = {11--15 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r15/main/assets/zinkevich17a/zinkevich17a.pdf}, url = {https://proceedings.mlr.press/r15/zinkevich17a.html}, abstract = {It is often asserted that deep networks learn “features”, traditionally expressed by the activations of intermediate nodes. We explore an alternative concept by defining features as partial derivatives of model output with respect to model parameters—extending a simple yet powerful idea from generalized linear models. The resulting features are not equivalent to node activations, and we show that they can induce a holographic representation of the complete model: the network’s output on given data can be exactly replicated by a simple linear model over such features extracted from any ordered cut. We demonstrate useful advantages for this feature representation over standard representations based on node activations.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Holographic Feature Representations of Deep Networks %A Martin A. Zinkevich %A Alex Davies %A Dale Schuurmans %B Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2017 %E Gal Elidan %E Kristian Kersting %F pmlr-vR15-zinkevich17a %I PMLR %P 401--410 %U https://proceedings.mlr.press/r15/zinkevich17a.html %V R15 %X It is often asserted that deep networks learn “features”, traditionally expressed by the activations of intermediate nodes. We explore an alternative concept by defining features as partial derivatives of model output with respect to model parameters—extending a simple yet powerful idea from generalized linear models. The resulting features are not equivalent to node activations, and we show that they can induce a holographic representation of the complete model: the network’s output on given data can be exactly replicated by a simple linear model over such features extracted from any ordered cut. We demonstrate useful advantages for this feature representation over standard representations based on node activations. %Z Reissued by PMLR on 04 October 2026.
APA
Zinkevich, M.A., Davies, A. & Schuurmans, D.. (2017). Holographic Feature Representations of Deep Networks. Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R15:401-410 Available from https://proceedings.mlr.press/r15/zinkevich17a.html. Reissued by PMLR on 04 October 2026.

Related Material