On Linear Identifiability of Learned Representations

Geoffrey Roeder, Luke Metz, Durk Kingma
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:9030-9039, 2021.

Abstract

Identifiability is a desirable property of a statistical model: it implies that the true model parameters may be estimated to any desired precision, given sufficient computational resources and data. We study identifiability in the context of representation learning: discovering nonlinear data representations that are optimal with respect to some downstream task. When parameterized as deep neural networks, such representation functions lack identifiability in parameter space, because they are over-parameterized by design. In this paper, building on recent advances in nonlinear Independent Components Analysis, we aim to rehabilitate identifiability by showing that a large family of discriminative models are in fact identifiable in function space, up to a linear indeterminacy. Many models for representation learning in a wide variety of domains have been identifiable in this sense, including text, images and audio, state-of-the-art at time of publication. We derive sufficient conditions for linear identifiability and provide empirical support for the result on both simulated and real-world data.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-roeder21a, title = {On Linear Identifiability of Learned Representations}, author = {Roeder, Geoffrey and Metz, Luke and Kingma, Durk}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {9030--9039}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/roeder21a/roeder21a.pdf}, url = {https://proceedings.mlr.press/v139/roeder21a.html}, abstract = {Identifiability is a desirable property of a statistical model: it implies that the true model parameters may be estimated to any desired precision, given sufficient computational resources and data. We study identifiability in the context of representation learning: discovering nonlinear data representations that are optimal with respect to some downstream task. When parameterized as deep neural networks, such representation functions lack identifiability in parameter space, because they are over-parameterized by design. In this paper, building on recent advances in nonlinear Independent Components Analysis, we aim to rehabilitate identifiability by showing that a large family of discriminative models are in fact identifiable in function space, up to a linear indeterminacy. Many models for representation learning in a wide variety of domains have been identifiable in this sense, including text, images and audio, state-of-the-art at time of publication. We derive sufficient conditions for linear identifiability and provide empirical support for the result on both simulated and real-world data.} }
Endnote
%0 Conference Paper %T On Linear Identifiability of Learned Representations %A Geoffrey Roeder %A Luke Metz %A Durk Kingma %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-roeder21a %I PMLR %P 9030--9039 %U https://proceedings.mlr.press/v139/roeder21a.html %V 139 %X Identifiability is a desirable property of a statistical model: it implies that the true model parameters may be estimated to any desired precision, given sufficient computational resources and data. We study identifiability in the context of representation learning: discovering nonlinear data representations that are optimal with respect to some downstream task. When parameterized as deep neural networks, such representation functions lack identifiability in parameter space, because they are over-parameterized by design. In this paper, building on recent advances in nonlinear Independent Components Analysis, we aim to rehabilitate identifiability by showing that a large family of discriminative models are in fact identifiable in function space, up to a linear indeterminacy. Many models for representation learning in a wide variety of domains have been identifiable in this sense, including text, images and audio, state-of-the-art at time of publication. We derive sufficient conditions for linear identifiability and provide empirical support for the result on both simulated and real-world data.
APA
Roeder, G., Metz, L. & Kingma, D.. (2021). On Linear Identifiability of Learned Representations. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:9030-9039 Available from https://proceedings.mlr.press/v139/roeder21a.html.

Related Material